Stefan Sundin

Hello YouTubers.

I’ve been looking for a way to get notified on new YouTube Live events somehow. I’ve been subscribed to the GData feeds for YouTube Live Events since Coachella, but it hasn’t worked that well since it’s not meant to be used directly by a feed reader for humans.

So I decided to code a quick PHP script that converts the feeds to something that is readable in most feed readers, where the entries link directly to the YouTube watch page of the event. As the GData API has four different feeds, featured, upcoming, live_now and recently_broadcasted, my script supports the same:

• youtube_live.php?id=featured (default)
• youtube_live.php?id=upcoming
• youtube_live.php?id=live_now
• youtube_live.php?id=recently_broadcasted

I find it interesting that the Google I/O keynote has been scheduled already, which will be live in 10 days and 17 hours: watch?v=jpQFnqpETy0. I’ll be there, will you? Let me know. =)

Another interesting thing in Feedly is that the service associates feed articles (the feed link) with likes on social media networks, so you will instantly get a measure on how popular an event is from a glance.

The featured feed only lists the latest 15 events, while the other feeds list 25 events, so some events that have completed may not be updated since they didn’t fit in the results. To fix this a backend service needs to keep tabs on events and check the event’s individual feeds if they have been updated, and so on. But this is only a quick hack and that would be too much work.

I’ve also posted the code for the script as a Github Gist: stefansundin/5519170. Put any suggestions either in the comments here or on the gist.

I hope you guys like it!

P.S. This may not be the time, but I want to add that XML processing sucks in most languages. SAX parsing is a pain and makes for ugly code (and it’s pointless if performance and size is not an issue). JavaScript had a small chance with E4X, but that has been abandoned by all browsers now. Bah. What you usually end up doing in these languages is that you just use regular expressions, which work surprisingly well. D.S.

Hi everyone.

One week ago I started a crowdfunding campaign in order to support the development for AltDrag 1.0, along with the release of beta 2. I hope that this crowdfunding campaign will bring new life to the project. It also gives users the opportunity to use their money to vote on the features and bugs they want me to spend the most time on.

I set the goal at $1000, and so far 17 people have pledged $280. At this rate the project should be funded before June 18th (the last day of the campaign). I suspect I might have to do something to keep it on track though, maybe another beta release.

This campaign is an easy way for users to give something back, and it gives them clear incentives to donate. The more people donate, the more time I will be able to spend on AltDrag, the better the next version will become.

Go read the full description at the fundraising site!

Coachella is streaming live on YouTube this weekend. It looks like a great show, and a lot of different artists are playing. I like one of the bands especially, Infected Mushroom. I decided to try to rip their show, and I was quite successful.

Many people use screen recording software to do this, but this approach have a lot of problems. The framerate is usually very uneven, and it has to encode it on the fly which takes a lot of CPU, the audio levels gets distorted because it passes through the mixer in your computer, and so on. The result is never that good.

Whenever possible, you should capture the stream and get the data files directly. To do that for YouTube Live, start by opening the YouTube page, like the current Coachella stream. Then use your favorite browser’s network monitor and watch for the entries that end with “index.m3u8?stream_seq=X”. These are requests for “playlist files” that contain urls to “shards”. The way YouTube Live works is by breaking the stream up into “shards”, which are then fetched individually by the clients and assembled during playback, and the playlist files contain several different urls (most likely used for load balancing). The size of the shards seems to range from 1 MB to 4 MB. What you need to figure out is what shard you want to start capturing from and what shard you want to stop at.

Note: The urls will only work for a few hours, and all urls in this post have already timed out! None of the commands here will work without a new url!

The requests you need to pay attention to.

If you right click the entry and copy the url, you will get something like this:

http://www.youtube.com/api/manifest/hls_playlist/id/z7ttfmzInwk.1/itag/96/source/yt_live_broadcast/ratebypass/yes/live/1/cmbypass/yes/newshard/yes/gir/yes/playlist_type/DVR/pmbypass/yes/maudio/1/cp/U0hVSlhQVV9KUUNONV9QSllJOkpaVjlSZVplT2pm/upn/CoU6Y0yLL4Y/fexp/932000,932004,906383,916911,916910,902000,901208,919512,929903,925714,931202,900821,900823,931203,906090,909419,908529,930807,919373,930803,906836,920201,929602,930101,930609,900824/sver/3/cpn/ZoiDPn4zGjWpdBen/ip/81.21.214.101/ipbits/8/expire/1365857857/sparams/ip,ipbits,expire,id,itag,source,ratebypass,live,cmbypass,newshard,gir,playlist_type,pmbypass,maudio,cp/signature/6AD4B2BFC126BFA177905B1B0A7ADFC0C36F4C71.4B7BA10E778BE86E2AA757EECD07CA99FB09E288/key/dg_yt0/file/index.m3u8?start_seq=5423

The parameter in the url that determine the quality is the itag value, where 92 is 240p, 93 is 360p, 94 is 480p, 95 is 720p and 96 is 1080p. You can’t just change it though, since the signature parameter will then become invalid. So pick the quality you want in the player and then copy the playlist url. When you have the playlist url, you can use mplayer to dump the stream for that shard. Use this syntax:

mplayer -dumpstream -dumpfile "stream.dump.5423.ts" -playlist "http://www.youtube.com/api/manifest/hls_playlist/id/z7ttfmzInwk.1/itag/96/source/yt_live_broadcast/ratebypass/yes/live/1/cmbypass/yes/newshard/yes/gir/yes/playlist_type/DVR/pmbypass/yes/maudio/1/cp/U0hVSlhQVV9KUUNONV9QSllJOkpaVjlSZVplT2pm/upn/CoU6Y0yLL4Y/fexp/932000,932004,906383,916911,916910,902000,901208,919512,929903,925714,931202,900821,900823,931203,906090,909419,908529,930807,919373,930803,906836,920201,929602,930101,930609,900824/sver/3/cpn/ZoiDPn4zGjWpdBen/ip/81.21.214.101/ipbits/8/expire/1365857857/sparams/ip,ipbits,expire,id,itag,source,ratebypass,live,cmbypass,newshard,gir,playlist_type,pmbypass,maudio,cp/signature/6AD4B2BFC126BFA177905B1B0A7ADFC0C36F4C71.4B7BA10E778BE86E2AA757EECD07CA99FB09E288/key/dg_yt0/file/index.m3u8?start_seq=5423"

Now you will get a file called “stream.dump.5423.ts”, a 5 second long clip that you can play with VLC. Anyone who’s reading this can figure out that all you have to do is create a simple script that dumps all the shards from shard X to Y, then just combine the files and you’re done. Here’s a bat script for Windows users:

set i=%1
set max=%2
if "%2" == "" (
	set max=%1
)
:loop
if %i% gtr %max% goto stop
if not exist "stream.dump.%i%.ts" (
	mplayer -dumpstream -dumpfile "stream.dump.%i%.ts" -playlist "http://www.youtube.com/api/manifest/hls_playlist/id/z7ttfmzInwk.1/itag/96/source/yt_live_broadcast/ratebypass/yes/live/1/cmbypass/yes/newshard/yes/gir/yes/playlist_type/DVR/pmbypass/yes/maudio/1/cp/U0hVSlhQVV9KUUNONV9QSllJOkpaVjlSZVplT2pm/upn/CoU6Y0yLL4Y/fexp/932000,932004,906383,916911,916910,902000,901208,919512,929903,925714,931202,900821,900823,931203,906090,909419,908529,930807,919373,930803,906836,920201,929602,930101,930609,900824/sver/3/cpn/ZoiDPn4zGjWpdBen/ip/81.21.214.101/ipbits/8/expire/1365857857/sparams/ip,ipbits,expire,id,itag,source,ratebypass,live,cmbypass,newshard,gir,playlist_type,pmbypass,maudio,cp/signature/6AD4B2BFC126BFA177905B1B0A7ADFC0C36F4C71.4B7BA10E778BE86E2AA757EECD07CA99FB09E288/key/dg_yt0/file/index.m3u8?start_seq=%i%"
)
set /a i+=1
goto loop
:stop

Save that as dump.bat and run it like “dump 5392 5989″ to download the whole Infected Mushroom set. Yes, you can do that even though that part of the Coachella event ended hours ago. My guess is that they allocate servers for this whole event and that the requests will work until the event ends and the servers are used for other purposes. This allows you to download the full event, from the very beginning, even though you start late.

You’ll get 598 .ts files that you have to join together. I suggest that you use TSSplitter for that. If you simply want to upload it to YouTube straight away, I suggest that you use mkvmerge to just mux the existing streams to an mkv container. This way you won’t transcode the streams and the quality will be the same, and it’s a lot faster than re-encoding. Even if you’re going to trim it (e.g. because you want it to start in the middle of a shard), I suggest that you use YouTube’s Video Editor, since that will skip an extra encoding step (YouTube uses the original file that you uploaded when you use the video editor).

I hope everyone enjoyed this little tutorial. If you did, please consider donating. Thank you!

To my surprise, a few days ago I discovered that I could no longer access any project on Google Project Hosting… I thought it was just an intermittent error and did not think more about it. However, days later I got the same error so I got suspicious. I logged out of my Google account and then I could access everything again. Or so I thought, ShutdownGuard was nowhere to be found.

I reported the error in a mailinglist and got the response that Google had banned my Google accounts since malware had been detected in one of my projects.  So apparently an automatic sweep detected malware in ShutdownGuard (presumably in one of the downloads) which resulted in the project being blocked and both my accounts banned. Now I know exactly what is falsely detected as malware. First, some backstory. Windows has really weird APIs that essentially gives the programs on the computer more control than the user itself. A program has the ability to tell Windows to shut down and other programs have the opportunity to stop Windows from shutting down, however, a program can also force the computer to shut down (it’s easy, just add EWX_FORCE), then no one can do anything about it. Windows is shutting down and there is nothing anyone can do about it (Windows Update forces shutdown in this manner). ShutdownGuard can stop the first type of shutdown, but not when a program supplies the force flag. To try to fix this, I created a companion program called ShutdownPatcher which injects itself into other processes in an attempt to rewrite their code so that they can not initiate a shutdown. On the website, the purpose and techniques of this program is well advertised, including the fact that it is highly experimental.

ShutdownGuard scheduled for deletion, and the wiki page describing the operation of ShutdownPatcher.

The response from Google has such a skeptical tone. It feels so uncomfortable to be suspected of spreading malware when they really have no proof. Automatic virus scanning with heuristics is such a flawed concept, the possibility of a detection being a false positives is just too great. I previously compressed my executables with UPX, a well known executable packer. It is so well known and useful that even malware and virus authors use it (in their case most likely because compressing the executable makes it more difficult to scan since the compression obscures the on-disk image), so in turn all UPX-compressed executables are detected as viruses in one way or another. Even though UPX-compression should be a perfectly valid option, due to this problem, anyone who uses UPX when releasing a product is crazy. When I release my products I always scan my final executables with VirusTotal, and I have on many occasions sent in samples to antivirus vendors so that they can whitelist my executables to prevent their heuristics from triggering false positives.

To be honest, this whole issue has made me want to move away from Google Project Hosting entirely. There has been many years since Google added something really useful to the site. If you look at their WhatsNew page, you quickly realize that the only real improvements they make to the site is to the issue tracker (which coincidentally is the only part that the Chromium project uses). But unfortunately moving is not so easy… since the ShutdownGuard project is still locked, I can no longer check out the code, and since it is still slated for deletion, I may lose my code forever. Google is essentially holding my code hostage, and at the same time preventing all my users – you guys – from making use of my program. You thought putting your code in the cloud would keep it safe? Think again.

I have hosted all my recent projects on Github, which by now is a far better and complete platform to host projects on. It has way more social features and it is a more collaborative environment. The only thing that Github lacks is a download area — they had one for a while, but they decided to remove that feature (I still wonder why). Just look at msysgit, which was previously completely hosted on Google, has now moved everything but the file hosting over to Github. I think it is fitting to compare Google Project Hosting to Google Video, which was really lacking in social features, it was too difficult to upload videos, it was even really difficult find videos, and it ultimately met its demise (I predicted this many years before it happened). It is also similar to Google Calendar and Google Reader, products that are no longer prioritized (and actually have big problems but are not considered important enough to be reworked). These are products that nowadays are merely maintained, and I think they will ultimately face their demise just as Google Video. There has been plenty of rumors that Google Reader is a products that’s constantly near cancellation.

To me, the worst part of this story is that I had to find out about it myself, I was not notified in any manner what so ever.  It is so typical of Google to not initiate contact, and even when in contact, the attitude of their employees is that it is my fault and that I should not expect due process. Now, Google has unbanned my account (but they only unbanned one of my accounts, how typical) but ShutdownGuard is still under review. I have no idea how long this can take. This is after all a support matter, and it should be dealt with swiftly. It is things like this that make people turn away from companies, and Google is no exception.

I got a nice little package in the mail a few days ago. AltDrag is featured in the German CHIP magazine 08/2012!

It’s only a small presence, but it’s there! Fun!