Xurui’s Blog

http-flv: getting started

Published On 十一月 25, 2017

category server | tags live flv rtmp

CONTENTS

Introduction

http-flv(HTTP FLV live stream) means delivery live stream in flv(flash video) format by http. It's a very common method to stream media seen in China because it has the advantage of low latency found in rtmp and easy delivery due to the use of http. For server side, rtmp stream is converted into flv because rtmp is almost the only method used to push stream now. For client side, playing flv stream is the same as playing flv video from a static server. Of course, a player that supports flv is needed.

advantages

  • has the same latency as rtmp(~3s)
  • no special protocol is required, much easier than rtmp
  • flv is widely supported except apple
  • support dns 302 redirect
  • not likely to be blocked by fire well

position of http-flv in video technique stack

techniques about video

  • codecs/compression
  • video player(client)
    • flv
    • others
  • video delivery methods(server)
    • Progressive Download
    • streaming(media streaming/live streaming)
      • RTSP/RTMP Streaming
      • Adaptive HTTP Streaming
      • http-flv

I am going to talk about video delivery methods → streaming → http-flv. And some skills about how to play flv will also be involved for debugging purposes. Video encode method such as H.264 is beyond the scope of this article.

Comparison between http-flv and other video delivery methods

Download

the end-user obtains the entire file for the content before watching or listening to it

Progressive Download

A progressive download is the transfer of digital media files from a server to a client, typically using the HTTP protocol when initiated from a computer. The consumer may begin playback of the media before the download is complete.

feature

  • the player starts video playback while downloading as soon as it has enough data
  • the file is downloaded to a physical drive on the end user's device, just like play a local file(wikipedia pointed out this feature is the key difference between streaming media and progressive download)
  • most widely used
  • easiest to implement: just put a video on your webserver and point your player to the URL
    • client: supported by Flash, HTML5 browsers, the iPad/iPhone and Android
    • server: only need a regular http webhoster that supports downloads
  • seek to a point not yet downloaded(pseudo-streaming) by doing range request
  • does not work for live

example

  • youtube
  • 优酷

implementation

video tag in html5 makes this extreamingly easy.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>video tag</title>
    <style type="text/css">
      video {
        width: 100%;
      }
    </style>
</head>
<body>
  <video controls="controls">
    <source src="movie.ogg" type="video/ogg">
    <source src="movie.mp4" type="video/mp4">
    Your browser does not support the video tag.
  </video>
</body>
</html>

Streaming media or media streaming

multimedia that is constantly received by and presented to an end-user while being delivered by a provider.

Live streaming is a special case in media streaming:

online streaming media simultaneously recorded and broadcast in real time to the viewer. It is often simply referred to as streaming

  • client: can not seek
  • server: stream is received constantly instead of generated from video file in disk

RTSP/RTMP Streaming

features

  • needs specialized webservers that only deliver the frames of a video the user is currently watching
  • has specific server(Wowza, nginx-rtmp) and protocol(RTMP) requirements
  • playback will experience interruption if the connection speed drops below the minimum bandwidth needed for the video
  • need streaming media type such as flv

example

  • 斗鱼

Adaptive HTTP Streaming

such as HLS, Dash

features

  • new
    • lack of standardization: hls vs dash vs others
    • support in HTML5 is currently under development
  • no special servers needed
  • storing your videos on the server in small fragments
  • long latency

example

  • 央视网
  • 虎牙

http-flv

besides RTSP/RTMP Streaming and Adaptive HTTP Streaming, http-flv is an unofficial protocol widely used in china.

features

  • use http: no special servers or protocols needed, a bit like progressive download
  • low latency(not worse than rtmp)
  • flv is easy to generate on the fly

example

  • 斗鱼
  • 熊猫tv
  • 虎牙
  • B站

This is a view of douyu(斗鱼) which is the most popular live website in china. douyu1.jpg douyu2.png

The distinctive character of http-flv is that there is a long connection with suffix of .flv. Generally speaking, a live website tends to use various streaming methods in different lines because there is no method that can fit all platforms and all application scenarios very well.

Difference between the 3 most common streaming methods:

+------------------+-------------------------------+-------------------------------+----------------------+
|     protocol     |             rtmp              |              hls              |       http-flv       |
+------------------+-------------------------------+-------------------------------+----------------------+
| full name        | Real-Time Messaging Protocol  | HTTP Live Streaming           | http flv live stream |
| proposer         | adobe                         | apple                         | -                    |
| transparent      | tcp                           | http                          | http                 |
| latency          | 1-3s                          | 5-10s                         | 1-3s                 |
| container format | flv                           | ts                            | flv                  |
| support          | flash player                  | native support on IOS&Android | flash player         |
+------------------+-------------------------------+-------------------------------+----------------------+

Implementation of http-flv

implementation is explained in 3 aspects and then followed by a simple implementation.

transport protocol

Here we focus on http. Content-Length header in http response indicates the length of body in bytes. HTTP clients will receive the specified length of data after parsing headers. There are 2 special cases where no Content-Length is provided:

  1. a Connection: close header is used to tell http clients to continue receiving data until the server side closes the connection
  2. use chunked encoding to transfer data between server and client in which case a header Transfer-Encoding: chunkedwill be provided. The server side will send an empty chunk to the client to indicate the end of the response.

The latter is better because the tcp connection can keep alive(useless in live scenario). Since chunked encoding is introduced from http1.1 so the latter can only be used in http1.1. The typical response headers of a http-flv request is like this:

curl --http1.0 -v http://192.168.3.234/testlive/game -o /dev/null
> GET /testlive/game HTTP/1.0
> Host: 192.168.3.234
> User-Agent: curl/7.49.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.5.7
< Date: Mon, 20 Nov 2017 09:18:48 GMT
< Content-Type: video/x-flv
< Connection: close
< Cache-Control: no-cache

or

curl -v http://192.168.3.234/testlive/game -o /dev/null
> GET /testlive/game HTTP/1.1
> Host: 192.168.3.234
> User-Agent: curl/7.49.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.5.7
< Date: Thu, 16 Nov 2017 03:54:49 GMT
< Content-Type: video/x-flv
< Transfer-Encoding: chunked
< Connection: close
< Cache-Control: no-cache

so the server side is responsible to set at least the following response headers:

Content-Type: video/x-flv
Connection: close
Cache-Control: no-cache

convert rtmp into flv(server)

rtmp is used to push stream to server and http is used by player to pull flv stream. Streaming servers that support http-flv must be able to convert rtmp stream into flv stream. Here are 2 streaming servers that support http-flv:

  1. ossrs/srs: It's a powerful live streaming server developed by Chinese.(You can see, Chinese have contributed a lot to live streaming)
  2. our hpms: The original nginx-rtmp-module doesn't support http-flv. Our customized version implements this feature in ngx_http_flv_live_module(HPMS/nginx-rtmp-module/hdl/ngx_http_flv_module.c).

rtmp

RTMP is aTCP-based protocol protocol designed for streaming video in real time. I am going to clarify some concept in rtmp protocol:

  • one conection contains several virtual channels(a channel for handling RPC requests and responses, a channel for video stream data, a channel for audio stream data, etc) on which packets may be sent and received,
  • packets(messages) from different streams/channels may then be interleaved, and multiplexed over a single connection
  • packet is fragmented into chunks

flv

FLV is the simplest video containing format. A flv video file consist of a header and a series of tags. A tag also has header and body. The header of a tag indicates this is a video or audio tag or script tag and what codec is used to encode the video or audio. The body of a tag is a frame of video or audio or other meta data. The video tags and audio tags are interleaved. This makes it possible to play the video from any point in a flv file.

When we push rtmp stream by flv format, the payload of a video/audio rtmp packet is the same as a flv video/audio tag body. It's obvious that the task of server side is:

  • receive rtmp stream
  • receive http request
  • parse rtmp packet header, extract the content of video/audio packet which represents a frame, pack it in flv tag, send it to client in http response body constantly.

The response body of http-flv request is just like a normal flv video. The server side sends the http response headers followed by the flv header(a constant in most cases). Then the server sends the tags generated simultaneously while receiving rtmp packets.

play http flv stream(client)

Most players that can play flv videos are able to play http flv live streams. It's a pity that apple doesn't support it because Jobs hated it.

  • players
    • vlc(all platforms)
    • potplayer(only available on windows)
  • browser(web)
    • PC
      • flv.js: a js library that parses flv file and pass the data into html5's video tag. This is a hack way to use h5 player.
      • video.js+flash tech: needs browser to support flash. Chrome is built with flash but safari has to install flash player plugin.
    • mobile
      • not possible
  • android&ios sdk

As far as I know, http-flv is mainly used in PC web live.

example 1: use flv.js

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Document</title>
    <script src="https://cdn.bootcss.com/flv.js/1.3.3/flv.min.js"></script>
</head>
<body>
    <video id="videoElement"></video>
    <script>
        if (flvjs.isSupported()) {
            var videoElement = document.getElementById('videoElement');
            var flvPlayer = flvjs.createPlayer({
                type: 'flv',
                isLive: true, // seems not necessary
                url: 'http://192.168.3.234:8080/testlive/game.flv'
            });
            flvPlayer.attachMediaElement(videoElement);
            flvPlayer.load();
            flvPlayer.play();
        }
    </script>
</body>
</html>

exmpale 2: use videojs+flash

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>videojs play flv</title>
    <link href="http://vjs.zencdn.net/6.2.8/video-js.css" rel="stylesheet">
    <script src="http://vjs.zencdn.net/6.2.8/video.js"></script>
    <script src="https://unpkg.com/videojs-flash@2.0.1/dist/videojs-flash.js"></script>

</head>
<body>
    <video
        id="my-player"
        class="video-js"
        controls
        preload="auto">
      <source src="http://192.168.3.234:8080/testlive/game.flv" type="video/x-flv"></source>
      <!-- <source src="rtmp://192.168.3.234/testlive/game" type="rtmp/flv"></source> -->
      <p class="vjs-no-js">
        To view this video please enable JavaScript, and consider upgrading to a
        web browser that
        <a href="http://videojs.com/html5-video-support/" target="_blank">
          supports HTML5 video
        </a>
      </p>
    </video>
    <script type="text/javascript">
        var player = videojs('my-player');
    </script>

</body>
</html>

A minimun implementation

below is a simple implementation(<200 lines) of a streaming server via http-flv based on python3's asyncio

import re
import logging

import asyncio
from aiohttp import web

streams = []
logging.basicConfig(format='%(asctime)s %(levelname)s %(filename)s:%(lineno)d: %(message)s',level=logging.INFO)

class Stream:
    def __init__(self, s_id):
        self.id = s_id
        self.flv_header = None
        self.meta_header = None # scriptData, AVC/AAC sequence header
        self.queue_size = 1000 # around 10MB for 720p
        self.tag_id = 0
        self.tags_queue = [None]*self.queue_size
        self.alive = False
        self.players = []
        self.condition = asyncio.Condition()

    def __repr__(self):
        return 'stream:%d' % self.id

    @property
    def player_num(self):
        return len(list(filter(lambda x:bool(x), self.players)))

    def end(self):
        # destroy the buffer when publish ended and number of players drops to 0
        if not self.alive and self.player_num == 0:
            logging.info('%s destroy the buffer' % self)
            del self.tags_queue

    async def push(self, reader):
        logging.info('push %s' % self)
        self.flv_header = await reader.read(9+4) # includes the size of tag 0
        if self.flv_header != b'FLV'+bytes([1,5,0,0,0,9,0,0,0,0]):
            logging.warning('stream is not flv')
            return
        while True:
            tag_header = await reader.read(11)
            if not tag_header: # eof
                break
            tag_size = int.from_bytes(tag_header[1:4], byteorder='big')
            tag_body = await reader.read(tag_size+4) # include the size of this tag
            tag = tag_header + tag_body
            logging.debug('%s receive tag:%d' % (self, self.tag_id))
            self.tags_queue[self.tag_id%self.queue_size] = tag
            # video/audio header sequence contains codec information needed by
            # the decoder to be able to interpret the rest of the data
            if not self.meta_header:
                if (
                    # Video tag not AVC sequence header
                    ((tag[0] & 0x1F == 9) and not ((tag[11] & 0x0F == 7) and (tag[12] == 0)))
                    or
                    # Audio tag not AAC sequence header
                    ((tag[0] & 0x1F == 8) and not ((((tag[11] & 0xF0) >> 4) == 10) and (tag[12] == 0)))
                ):
                    logging.info('%s first media tag at %d' % (self, self.tag_id))
                    self.meta_header = b''.join(self.tags_queue[:self.tag_id])
                    self.alive = True
            self.tag_id = self.tag_id + 1
            with await self.condition:
                self.condition.notify_all()
        logging.info('%s closed, %d tags received' % (self, self.tag_id))
        self.alive = False
        # in case some players are waiting next tag
        with await self.condition:
            self.condition.notify_all()
        self.end()

    async def pull(self, resp):
        player_id = len(self.players)
        player = 'player:%d' %  player_id
        self.players.append(True)
        resp.write(self.flv_header)
        resp.write(self.meta_header)
        find_key_frame = False
        offset = self.tag_id - 1
        logging.info('%s %s pull from offset %d' % (self, player, offset))
        while True:
            if offset == self.tag_id:
                if not self.alive:
                    self.players[player_id] = False
                    self.end()
                    return
                try:
                    await resp.drain() # send data here
                    if not self.alive:
                        # won't be notified anymore so don't wait
                        continue
                    logging.debug('%s %s wait for more tags' % (self, player))
                    with await self.condition:
                        await self.condition.wait()
                except asyncio.CancelledError:
                    logging.info('%s %s disconnected' % (self, player))
                    self.players[player_id] = False
                    self.end()
                    raise
                continue # important
                # continue
            elif offset < self.tag_id - self.queue_size:
                logging.info('%s %s is behind' % (self, player))
                offset = self.tag_id - 1
                continue
            # offset in [tag_id-queue_size, tag_id)
            tag = self.tags_queue[offset%self.queue_size]
            if not find_key_frame:
                # video tag and key frame
                if (tag[0] & 0x1F == 9 and (tag[11] & 0xF0) >> 4) == 1:
                    find_key_frame = True
                    logging.info('%s %s find keyframe tag:%d' % (self, player, offset))
                else:
                    offset = offset + 1
                    continue
            logging.debug('%s %s send tag:%d' % (self, player, offset))
            resp.write(tag)
            offset = offset + 1


async def handle_pull(request):
    m = re.search(r'/(\d+)', request.path)
    if m:
        stream_id = int(m.group(1))
        if stream_id < len(streams):
            stream = streams[stream_id]
            if stream.alive:
                resp = web.StreamResponse(headers={'Content-Type': 'video/x-flv', 'Connection': 'close', 'Cache_Control': 'no-cache'})
                if not (request.version[0] == 1 and request.version[1] == 0):
                    # mplayer use http1.0
                    # use chunked encoding for http1.1 and later
                    resp.enable_chunked_encoding()
                await resp.prepare(request) # sender headers
                await stream.pull(resp)
                return resp
    return web.Response(status=404, text='stream does not exit')

async def handle_push(reader, writer):
    """callback for client connected
    args: (StreamReader, StreamWriter)
    """
    stream = Stream(len(streams))
    streams.append(stream)
    await stream.push(reader)
    writer.close() # close socket

loop = asyncio.get_event_loop()

# stream server
tcp = loop.run_until_complete(asyncio.start_server(handle_push, '0.0.0.0', 8888, loop=loop))
# http server
server = web.Server(handle_pull)
http = loop.run_until_complete(loop.create_server(server, '0.0.0.0', 8080))

# Serve requests until Ctrl+C is pressed
logging.info('start...')
print('''
push address tcp://{}:{}
pull address http://{}:{}'''.format(*tcp.sockets[0].getsockname(), *http.sockets[0].getsockname()))
try:
    loop.run_forever()
except KeyboardInterrupt:
    logging.warning('stop...')
    pass

# Close the server
# close listening socket, current requests are still procedding
tcp.close()
loop.run_until_complete(tcp.wait_closed())

# loop.run_until_complete(server.shutdown()) # close transports
http.close()
loop.run_until_complete(http.wait_closed())

# cancel all pending tasks
tasks = asyncio.Task.all_tasks()
for task in tasks:
    task.cancel()
results = loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
print(results)

loop.close()

publish 

ffmpeg -re -i /opt/yxr/game.flv -c copy -f flv  tcp://192.168.3.234:8888

play 

http://192.168.3.234:8080/<stream id>

stream id starts from 0. The first stream published is 0, the second stream published is 1, and so on.

load test

Make sure to restart server before load tets.

publish.sh
function output() {
    for i in {1..100}
    do
        echo " -c copy -f flv tcp://127.0.0.1:8888"
    done
}

# publish 100 streams from local
ffmpeg -loglevel warning -re -i /opt/yxr/game.flv $(output) &

# tpo output in batch mode of the server process &ffmpeg
top -b -n 3 -d 30 -p $(pgrep -f streamming_server.py),$(pgrep ffmpeg)

pkill ffmpeg

output 

> sh publish.sh
top - 12:57:57 up 96 days,  1:23,  2 users,  load average: 0.11, 0.12, 0.08
Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.1%sy,  0.3%ni, 99.2%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  148691916k total, 97583768k used, 51108148k free,   751296k buffers
Swap: 67108860k total,    28580k used, 67080280k free, 91219068k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17724 root      20   0  233m  26m 4688 S 67.9  0.0   0:00.77 python
17756 root      20   0 68336  20m 3424 S 14.0  0.0   0:00.11 ffmpeg


top - 12:58:27 up 96 days,  1:24,  2 users,  load average: 0.47, 0.20, 0.11
Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.7%us,  0.5%sy,  0.0%ni, 96.5%id,  0.1%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  148691916k total, 97968144k used, 50723772k free,   751296k buffers
Swap: 67108860k total,    28580k used, 67080280k free, 91219076k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17724 root      20   0  606m 399m 4868 R 71.2  0.3   0:22.14 python
17756 root      20   0 68224  21m 3444 S 10.2  0.0   0:03.16 ffmpeg


top - 12:58:57 up 96 days,  1:24,  2 users,  load average: 0.48, 0.23, 0.12
Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.7%us,  0.5%sy,  0.0%ni, 96.6%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  148691916k total, 97964160k used, 50727756k free,   751296k buffers
Swap: 67108860k total,    28580k used, 67080280k free, 91219084k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17724 root      20   0  605m 399m 4868 R 70.7  0.3   0:43.36 python
17756 root      20   0 68568  22m 3444 S 10.0  0.0   0:06.15 ffmpeg

play.sh
# publish a stream from local
ffmpeg -loglevel warning -re -i /opt/yxr/game.flv -c copy -f flv tcp://127.0.0.1:8888 &

# top output of the server process
top -b -n 3 -d 30 -p $(pgrep -f streamming_server.py) &
pid=$!

# in case 404
sleep 0.1
# pull the stream from 100 players
wrk -t 1 -c 100 -d 60 http://127.0.0.1:8080/0 > /dev/null

wait $pid
pkill ffmpeg

output 

> sh play.sh
top - 13:57:51 up 96 days,  2:23,  2 users,  load average: 0.03, 0.05, 0.01
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.1%sy,  0.3%ni, 99.2%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  148691916k total, 97566692k used, 51125224k free,   751296k buffers
Swap: 67108860k total,    28580k used, 67080280k free, 91214296k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
22042 root      20   0  228m  22m 4868 S 39.9  0.0   0:00.62 python


top - 13:58:21 up 96 days,  2:23,  2 users,  load average: 0.28, 0.11, 0.02
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.8%us,  0.7%sy,  0.0%ni, 96.1%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  148691916k total, 97571888k used, 51120028k free,   751296k buffers
Swap: 67108860k total,    28580k used, 67080280k free, 91214304k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
22042 root      20   0  232m  25m 4868 R 80.3  0.0   0:24.73 python


top - 13:58:51 up 96 days,  2:24,  2 users,  load average: 0.49, 0.17, 0.05
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.8%us,  0.7%sy,  0.0%ni, 96.1%id,  0.1%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  148691916k total, 97569968k used, 51121948k free,   751296k buffers
Swap: 67108860k total,    28580k used, 67080280k free, 91214316k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
22042 root      20   0  231m  25m 4868 S 81.1  0.0   0:49.07 python
It shows this simple server can handle 100 streams at the same time.

Future about FLV

we are talking about http-flv here, but flash/flv has many disadvantages so that it's not supported by apple even adobe has given up developing it. Html5 is replacing flash and mp4 is replacing flv. The abandon of flash has achieved great progress in countries except China. So that, mobile browsers do not support Flash, and modern desktop browsers make it increasingly difficult to use Flash or disable it by default. However, is mp4 suitable for live streaming? No, because mp4 is not seekable. The biggest advantage of flv is that it's a seekable which means it can start to play from any position in the file. That's why there's no http-mp4. That's why when we use ffmpeg to push rtmp stream, we should specify format as flv(-f flv). In a word, flv is still the main container format used in live streaming so far. But it will be replaced some day.

Reference

« 集群环境下安装python应用最佳实践 查看linux的配置 »
qq email facebook github
© 2024 - Xurui Yan. All rights reserved
Built using pelican