Video Artificial Intelligence Powers Creation

The digital video industry has major challenges with creation because editing and production costs are rising. Lack of video inventory is universal; however, there is hope in leveraging video artificial intelligence (VAI) to address these problems. Massive amounts of live and pre-recorded video never reach the audience. Publishers are being asked to do more with less. Publishers are hamstrung due to small editorial budgets and shifting priority while massive video revenue goes untapped. The need to intelligently create video inventory couldn’t come fast enough to feed the insatiable mobile video appetite. Here we dig in on the What, Where and Why in VAI and rethink how content is being created, edited and delivered all towards building greater video lifetime value.

Above video was created using video artificial intelligence. Animated preview thumbnail selected using machine learning.

Rise of Video Artificial Intelligence

Automation is not the cure-all; however, combining live streams and artificial intelligence solves several pain points while augmenting the production staff and delivering relevant content. Your editorial team is the most valuable resource in your video workflow. This is the very reason why these highly-valued editors should be focused on higher value video creation. VAI isn’t about replacing your editorial team but scaling lower priority content at a fraction of the cost while increasing revenue generation.
Object detection YOLO Basketball labeling actions MicroClips InfiniGraph
There are many promising video AI examples showing off real-time facial tracking, object recognition and image labeling etc. In our previous post Top Video Artificial Intelligence and Machine Learning at NAB 2018, we highlighted several early AI products in the market. Outside of the video labeling services several are extending existing video editing systems as a cloud Saas. So what does this all mean for my existing video assets? First off, there has been monumental advancements in algorithms to obtain a deeper understanding of actions people take within a video. Such as YOLO and Deep Structured Models to label individual actions in video. This in-depth dissection changes the game of measuring what parts of a video resonate with consumers vs. old linear video measurement which woefully lacks meaningful insights.

Promising Use Cases

Using VAI to create and edit video isn’t new. Companies Like Adobe and IBM are all using advanced video/image analysis to enable smarter editing platforms. However, publishers need more than just editing assistance they need scalable video creation. One of the more profitable use cases is short video highlights and previews along with extending in context video compilation. These combinations have proven to be highly lucrative and well suited for VAI in both creation and scale.



Example above demonstrates using video artificial intelligence to create and control audio and video transitions between clips. The video length and segments being used are determined based on video scoring generated via VAi.

Another exciting use case is making video libraries searchable. Finding specific people, actions, scenes within video opens the doors to advanced video discovery. A whole new world opens up with amazing possibilities of cross-reference and extracting information otherwise trapped in old school liner video metadata.

The Opportunity

Solving the problem of creating net new video from existing video assets opens up a monster revenue source while extending your overall video lifetime value. Taking long-form video or live content and compressing to meaningful short-form mobile-ready is a game-changer. VAI creation does not require expanding editorial budgets or delaying critical production windows.
Artificial Intelligence Video Scoring InfiniGraph MicroClips
Organizing content to be contextually relevant will always be a key factor in ensuring meaningful content flow. What clip’s and scenes logically flows together are subjective unless you are directly measuring audience response. This is why utilizing an active feedback mechanism to improve relevance is important. Here we address some top challenges,

Top challenges

  • Context
  • Time availability
  • Quality
  • Shared viewing experience
  • Interactive storyline


  • Football search terms compilation highlights most searchedThere isn’t a lack of challenges facing video creation at scale. For sure content specifically created for social platforms that harness the human connection and social dynamics will produce a higher quality product. However, how do you scale this type of creation? Combining AI with video provides the opportunity to develop a video learning system to assist in delivering quality output that builds on itself over time. Hence, why a learning system is ideal for video creation based on reproducible tasks.

    Introducing MicroClips

    Here we describe the strategy of video “fracking” your entire library. The ability to organize video libraries, extract cross-referenced videos, annotate video content and create relevant net new video makes MicroClips possible. Manipulating video at scale using artificial intelligence creates an enormous revenue opportunity. The ability to learn what works and adjust video assets requires thinking differently about how content is produced. The possibilities are endless over many content types, moreover, digital enables measuring levels of audience engagement like never before.



    Above is a sports MicroClips compilation created using AI and is derived from many videos. MicroClips are created using incontext high value action sequences. These clips are places together to create the MicroClips compilation. What’s exciting is the variety of compilation that can be created and watching what the audience finds most entertaining. Highlights and compilations have garnered the highest play and share rates. Some have even “gone viral” !
    InfiniGraph MicroClips compilation Artificial Intelligence created video
    Want to see more exciting Sports MicroClips in action?

    Endgame

    The ultimate goal is to provide a cost-effective video creation process while improving viewability, quality and consumer play rates. Leveraging how consumers connect and the time they have available are additional driving factors to a smarter creation process.
    There are many big bets being made as well as major networks investing in dedicated Snapchat teams to capture a slice of this multi-billion dollar pie and prove the model out.

    Conclusion

    Quality content and great storytelling require a human touch for now but for large-scale video organizing and creation, the future is bright for video artificial intelligence. The digital video industry is going through a major transformation and the consumers are the winners. We have seen the networks being put on notice with the incumbents not only producing top shows, receiving awards, drawing in compelling audience sizes but also leveraging advanced technology faster improving the consumer experience. The publishers that harness the competitive advantage of VAI creation will deliver quality video to market faster with higher relevance all resulting in greater consumer retention and revenue.

    Top Video Artificial Intelligence and Machine Learning at NAB 2018

    Video artificial intelligence was a massive theme at NAB 2018 with a majority of the video publishing technology companies showing off some form of AI integration. In my previous post How Artificial Intelligence Gives Back Time time is money in the video publishing business. AI is set to be a very important tool why all the big guns like AWS (Elemental), Google (Video Intelligence), IBM (Watson) and Microsoft (Azure) had digital AI eye candy to share. There was a feeling of a meet too with all of them were competing to weave their video annotation/labeling – speech to text APIs into a variety of video workflows.

    Top Video AI use cases:

    1. Labeling – The ability to label the elements within a video specific scene selection, people, places, and things.
    2. Editing – Segmenting by relevance, slicing up the video into logical parts and producing.
    3. Discovery – Using both annotation and speech to text to expand metadata for funding specific scenes within video libraries.

    Challenges

    One of several challenges is this ALL or nothing situation. Video publishers assets can be on many hard drives or encoded without lots of metadata. There are companies that provide services like Axel to index those videos and make them searchable with a mixed model of on-prem tech and cloud services. Dealing with live feeds requires hardware and bigger commitments. Most publishers are not willing to forklift over their video encoding and library to another provider without a clear ROI justification. The other big ROI challenge is video publishers don’t have a lot of patience and the pressure to increase profits on video is higher now with more competition in the digital space across all channels. Selling workflow efficiency in AI won’t be a big enough draw over AI generating substantial revenue solving a specific problem. The pain isn’t high enough to make a big AI investment. There are lots of POC right now in the market, however, not one product creates a seamless flow within a video publisher’s existing workflow. Avid and Adobe are well positioned for the edit team since their products are so widely used. Other cloud providers are enabling AI technology not a specific solution.

    AI Opportunity

    Search and discovery was the biggest theme using AI to do image and speech to text analysis. Compliance with Closed Caption to make video accessible in digital will be mandated driving faster adoption. Editing video via AI is in its early phase, however, the technology is emerging fast. There are some exciting examples of AI created video but at scale is another. Of the many talks at NAB some exciting direction on AI in Video were discussed around video asset management. Here are a few examples of what we demoed at NAB 2018 showing promise in the video intelligence field.

    Adobe Sensi

    Adobe Video SegmentingAdobe had a big splash with their new editing technologies and using AI to enhance the video editing process. Todd Burke presented Adobe Sensi their AI entry into video intelligence. The video labeling demo and scene slicing we’re designed to help editors create videos faster and simplify the process. The segmenting was just a prototype and video labeling demonstrated the API extension integrated within Sensi. Adobe Labeling Demo

    IBM Watson

    IBM Watson Video SegmentingIBM’s demo was slick and pointed to the direction of using machine learning to process large amounts of video to pull out interesting parts of the video. Doing the announcer and crowd response analysis added another layer of segmentation. You can see a live demo of their AI highlight for the Master. They did the same for Wimbelton slicing up the live feed they were powering for the events and creating “cognitive highlights”. It wasn’t clear if these highlights were used by the edit team or if this was a POC. Regardless there was both image and text analysis of the steams occurring and demonstrated the power of AI in the video.

    Avid

    Avid Video analysisThe Avid demo was just that. They created a discovery UI on top of API’s like Google Vision to assist in the video analysis for search and supporting edit teams. Speech to text and annotation in one UI has its advantages. It’s wasn’t clear how soon this was going to be made available over a development tool. Avid Labeling

    Google Vision

    Google Vision Zoro labelingThe team over at Zora had by far the slickest video HUB approach. I believe the play for Google is more around their cloud strategy trying to attract storage of the videos and leverage their Video Intelligence to enable search over all your video assets. Google’s video intelligence is just getting started and their opening up of the AI foundation Tensorflow makes them one of the top companies committed to video AI. I like what Zora is doing and can see editing teams benefiting from this approach. There was a collaborative element too.

    Microsoft Azure

    Azure GreyMeta2GrayMeta UI was slick and their voice to text interface was amazing. This was all powered by Azure. Azure Video Indexer is the real deal and ability to identify face identification has broad use cases. Indexing voice isn’t new but having a fast and slick UI  helps enable adoption of the technology. They can pinpoint parts of the video just on the text along. There is a team collaboration element around the product have a Slack feel. The approach was making all media assets searchable.

    AWS Elemental

    There were several cool examples of possibilities with Amazon Rekognition - video analysis, facial recognition and video segments. Elemental (purchased by Amazon) core technology is a video ad stitching whereby video ads are inserted into the video directly. They created UI extension demonstrating some possibilities with Rekognition.  It wasn’t clear what was in production over the demo. The facial recognition around celebrities looked solid. AWS Singular Analysis Tracking PeopleElemental had a cool real-time object detect bounding boxes showing up on sports content too. This has many use cases, however, creating more data for video publishers to access when the amount of data they can manage needs to be addressed before adding another data firehosed. AWS Elemental label celebrity words SM

    Conclusion

    Video artificial intelligence is just getting started and will only improve with greater computing advancements and new algorithms. The guts of what’s needed exist to achieve scale.  The major use cases around video discovery and search are set to improve dramatically with industry players opening up more API’s. Video machine learning has great momentum utilizing these API’s to crack open the treasure trove of data locked away inside of video. The combination of video AI and text analysis truly creates a massive metadata for the multitude of use cases where voice computing can play are roll. Outside of all the AI eye candy there needs to be more focus on clear business problems vs. Me Too. More like what’s the end product and how will it make the video publisher more revenue?

    How Artificial Intelligence Gives Back Time

    How AI – Artificial Intelligence and machine learning gives back time? It’s not a secret that AI is here and coming much faster than many other technology booms. Some are saying we’re in the 3rd wave of computing. In our previous post 3 Ways Video Recommendation Drives Video Lifetime Value we talk about how machine learning is transforming finding and recommending videos to enhance consumer experience. For me, I’m just excited to be part of the machine learning business and creating powerful products focused on improving the digital video experience. I was recently commissioned to put together a short video and deck on how Artificial Intelligence is transforming the device-based experience for consumers. As part of this project the brands we’re looking to understand the different ways AI is going to potentially impact their business. Here were the main topic areas.

    • How do you stay ahead of where AI is headed?
    • How should AI be leveraged to enhance brand trust, improve engagement and help consumers get jobs to be done in a way that is valued by consumers?
    • How can AI be employed to create better personal performance for individuals?

    The presentation was to top brands like Scripps Network, Hertz, Bacardi, Planet Fitness, Arizona State University and DX Marketing.

    Video Transcript ….

    Hi I’m Chase McMichael CEO and co-founder of infiniGraph. InfiniGraph focuses on increasing video lifetime value for video publishers and broadcasters and we do that through processing vast amounts of their video data and understanding what visual in the video engage consumers. By measuring what image or video clips are most engaging within certain scenes we’re able to increase video consumption showing the right images or clips to excite the consumer. Here’s a great example of a video clip that we extracted out of a video for CBS. By putting this specific clip in front of the right person at the right time we’re able to dramatically increase video take rates as produce a better consumer experience.

    I was asked to talk about machine learning / artificial intelligence and how it would affect brands and improve consumer experience around devices. One of the exciting things about artificial intelligence primarily in my point of view is AI will give back time to individuals. AI is about making smart decision for them or providing insights proactively.

    So how should AI be leverage to increase brand trust, engagement and help consumers?First off brand trust, brand trust is about anticipating your consumers, being able to be very proactive when they interface with you. It’s important to actually recognize them and provide them with incredible value and service. This is something that all brands have struggled with. Its not you know someone comes into a retail establishment or comes online but laking responsive is lost opportunity. A big opportunity is personalization. The ability to personalize ones experience is a big deal right now. Companies are really utilizing their data in some smart ways especially in the retail segment we’re seeing this with Amazon and a lot of the movie companies like Netflix trying to customize the experience for their audience.

    The other thing that we’re seeing around brand trust is the ability to really not only be intuitive and responsive but proactive. Being proactive requires a much higher level of intelligence around your data. Taking that data to the next level of insights where you’re really thinking is AI. The key is anticipation like what does that consumer going to buy or how are they going to respond? When they purchase a product being more proactive creates an incredible experience. Again brand trust is easy to lose.  Brands spend many many decades or a hundred years on creating trust and all of a sudden something happens and the internet revolts. They become completely eviscerated.

    Its critical brands are responsive to what’s happening across the social web and monitoring intelligently. How you interface with your consumers across mobile, social or over cloud application all requires intelligence.

    Don Peppers back in their late 90s was doing what’s called one-to-one marketing and this is really the onset of personalized experience. If you’re going to enhance consumer engagement you really have to put in front of that consumer something that visually and cognitively gets them excited. Are you creating an emotional response? Without emotion people do not recall information so if you don’t make an emotional response with someone the ability for them especially in this distracted economy to engage approaches zero.

    Getting consumer to recall is very difficult so good service is expected the reality is people want to be wowed and that’s really comes down to how well your consumer touch points are response and intuitive.  Do you know about the individual interacting with you on a day to day or month to month or year to year basis? A core component to any company is understanding your consumers and their individual behavior. A customer interacts with your brand do you have the ability to recommend or provide insights that helps and again give them back time? If not you’re really have done them a disservice. Your interface has to be fluid or you create drudgery. 

    Improving engagement is the big win with artificial intelligence. System designed to predict and be intuitive through giving back time will lead their industry.  The core components that any enterprise must think about is how their enterprises is going to re-engineer around consumer data as well as capturing data to drive an active feedback loop. This feedback mechanism between the consumers and that touch point will be the foundation of an AI system.

    Help consumers and creating a frictionless environment will win your consumer over. Driving proactive actions that actually work. The other question you have to ask yourself is how are you utilizing that information to create a very robust profile so that you’re actually having a conversation with your customer.

    You actually know a lot about their history and you know a lot about what was successful. Start engaging with you consumer by understanding their product usage and leverage that to information to improve the experience.  From an AI perspective, now you have a system out there mining data to surface functional clusters of information both visually maybe vocally as well as across just standard data sets.

    Think big here because we’re now in such a connected community in society that with a push of a button I can share a picture with thousands of people and then the whole conversation spurts up around maybe your product being both positive and negative sense. How are you inserting yourself in that conversation proactively is very important.

    How do you physically help a consumer really comes down to can you be intuitive on their needs. Think about personal data assistant or personal AI assistance. Personal assistants are going to be very intuitive very smart and crawl lots of different data sources. These AI assistance are proactively telling you to do thing to simplify your life. An example is let them go out and find this information for you. These types of digital assistants are gonna be extremely important in people’s lives because now things that they had to spend their mental time on they will be spending more time on more higher cognitive thinking than the mundane check off tasks that we do today.

    If a brand is able to give time back to there consumer creating an intuitive and very frictionless experience will be the go to experience. Now your brand will dominant with that customer because they’re going to create not only the loyalty clearly but creating engagement through helping that consumer.

    Another areas that is creating lots of buzz is artificial intelligence taking over jobs. Clearly you as a big industry or brand don’t want to become the job killer in the industry and that’s a big issue for executives thinking about implementing intelligent automation. Everyone is reading everywhere that the robots are going to take over the world the reality is how are you augmenting your staff. What can you do to enable your staff to be more intelligent utilizing internal resources to be more efficient and effective when they interface with consumers.

    The real crux in this whole equation is how to enhance that experience with consumers and be able to empower my employees to be smarter and faster while creating a symbiant relationship.

    Another thing around artificial intelligence is you really need to be thinking not in quarters but in decades. The companies that have really focused on digital transformation and utilizing data intelligence to transform their business will be the disruptors. Are you going to be the leader, because those that have executed AI in their business will have the speed and ability to adapt enabling them to trump anything that comes to the table. Think AI first.

    A picture is worth a thousand words so in my business videos are worth tens of thousands of words and for us we’re wanting to find that unique image or video clip within a video segment or even a long video formats that really gets consumers excited and gets them engaged. This visual intelligence is critical in my business and very important for many brands. Using visual intelligence especially in video for marketing is an incredible opportunity. What you put in front of your consumers and can learn from that engagement around those visual properties within the actual images themselves is insight. The ability to adjust video is a competitive advantage creating a higher order of thinking where you’re giving a machine the ability to transform content in real-time. That’s that artificial intelligence we talked about previously so really being able to think about how do I take my brand in my industry and start thinking about all the data that’s coming in.

    Its all about data in and quality and consumer engagement out.