Flight tracking via ADS-B on a Raspberry Pi

Here's a fun little project you can build that is at the cross-roads of computing and radio communications: a flight tracker using SDR (Software-defined radio) to receive ADS-B transmissions directly from aircraft flying overhead using a Raspberry Pi with a DVB-T USB stick. Once you've built the system, you can direct your web browser to a port on the Raspberry Pi to take a look at all the airplanes near your location - not a lot going on above the White Mountains this Sunday afternoon:

Screenshot 2014 12 28 14 19 05

I've recently built three of these and linked them all to the FlightAware tracking website, where you can see the flights currently tracked by all three receivers, as well as tracking statistics regarding number of flights seen per day, etc. Here is the complete system with the three cables being power, Ethernet, and the antenna connection:

20141228 151553 1

The basis of this tracking is the Automatic dependent surveillance – broadcast (ADS–B) that each airplane transmits on a frequency of 1090 MHz, which contains GPS position, speed, altitude, heading, ascent/descent, and other navigational data. This information is normally used by ATC (Air Traffic Control) as well as received by other airplanes to provide situational awareness and allow self separation i.e. collision avoidance. Since this is being broadcast in a standardized format, it can be received and decoded by anybody, including ground stations.

Which brings us to the cheapest and most interesting way to receive these signals: SDR, or Software-defined radio - a technology where components that have been typically implemented in hardware (e.g. mixers, filters, amplifiers, modulators/demodulators, detectors, etc.) are instead implemented by means of software in a computer. All you need is an antenna and a UBS device that can support SDR applications, such as a cheap DVB-T USB stick.

For the computer system we don't need much processing power, so the Raspberry Pi Model B+ is the perfect choice for a low-priced stand-alone system that can easily handle the decoding of the ADS-B signal using the open-source dump1090 software and stream the data to a tracking site, such a FlightAware.

FlightAware also has some good instructions on how to build the system as well as a shopping list of all the components you will need: Build a PiAware ADS-B Receiver. Overall, the complete system, including case, power supply, Ethernet cable, etc. will cost you about $105.

However, the tiny antenna that comes standard with the DVB-T stick is only good for reception of signals from a very limited range. So one of the components you might want to upgrade sooner or later is the antenna by getting one that is actually appropriate for 1090 MHz, such as this vertical ADS-B outdoor base station antenna, or this ADS-B blade indoor antenna. I opted for the indoor antenna, since I didn't want to run extra antenna cables to the roof. And the indoor antenna is already so much better than the original tiny DVB-T whip.

Screenshot 2014 11 19 10 44 58

As you can see in this diagram, upgrading the antenna on November 12th resulted in the system being able to receive about 40,000 - 50,000 positions per day instead of 13,000 - 14,000 positions with the tiny original antenna - the correct antenna really makes a huge difference in the capability and range of the system!

Overall this is a fun little project to do on a rainy weekend. You can either build it all by yourself, or use it as an opportunity to teach the kids how to build a computer. Some Linux and networking skills are required, but nothing too complicated. And there are good instructions available for each step of the process.

Extracting useful data from HTML pages with XQuery

When building in-house solutions or mobile enterprise applications, you are often faced with having to deal with legacy systems and data. In some ancient systems, the data might only be available as CSV files, in other cases it might be arcane fixed-length text reporting formats, but if the legacy system is less than 20 years old, chances are pretty good that someone built and HTML front-end and so the data is available through a browser interface that renders it in some poorly formatted HTML code that loosely follows the standard. And very likely you will find the data intermixed with formatting and other information, so extracting the useful data is usually not as easy as it sounds.

In addition, when you are building mobile solutions, you may sometimes need some government data that is not yet available in XML or another structured format, so you again are faced with having to extract that information from HTML pages.

Common approaches to extracting data from HTML pages, such as screen-scraping and tagging are cumbersome to implement and very susceptible to changes in the underlying HTML.

In this video demo I want to show you a better way of extracting useful and reusable data from HTML pages. In less than 15 minutes we will build a mobile solution that - as an example - takes Consumer Price Index data from the US Bureau of Labor Statistics, parses and normalizes the HTML page, and then uses an XQuery expression to build nicely structured XML data from the HTML table that can then be reused to build a CPI chart. I will walk you through the creation of the XQuery expression step-by-step so that you can easily apply this method to similar problems of HTML data extraction:



As you can see in the above video, it was fairly easy to create nicely structured XML data from a table in the HTML page and to create a first simple chart that plots the CPI data over time.

But the true power of this approach is that you have much more flexible charting capabilities in MobileTogether and the XML data is now reusable, so you can calculate annual inflation rates directly from the underlying CPI data and plot it as well.

In this next video demo I want to show you just how to do that in less than 10 minutes. We will add a year-range selector to our chart where we can define which years to plot, and we will add an overlay chart that derives the annual inflation rate based on the underlying CPI data using XPath calculations and the plots that data:



Using this technique, you can not only extract data from singular HTML pages, but easily build a modern mobile front-end experience for many legacy systems that just offer an HTML-based browser interface at present. This will enable you to make your workforce a lot more productive and efficient, as they can now use a friendly mobile app experience to access your system rather than having to deal with a couple of HTML pages and forms in a browser on their tiny smartphone screens.

moto 360 Review

A while ago I wrote about my somewhat disappointing experience with the original Galaxy Gear, Fitbit, and Google Glass in an article "The (Broken) Promise of Wearables". It seems that I may have to revise my opinion a bit based on the new moto 360 smart watch:


First of all, I will admit that I'm a huge watch aficionado and have a collection of several beautiful mechanical timepieces and complications, as well as functional sport watches. So the Galaxy Gear  just hurts from a design perspective - both in its original form as well as the Gear 2 and the new Gear S. I have also been less than impressed by the new Apple Watch. Despite all the claims by others that it is beautiful, in my eyes it has the same flaw as the Galaxy Gear: the watch is square. Most display screens are rectangular, so they just built a watch around a square or rectangular screen.

However, there is a reason that the majority of watches have evolved with a circular dial. It is the most comfortable to wear, because it doesn't limit the movement of your wrist. And it has a timeless elegance to it.

So I was actually quite excited to receive my moto 360 this week and give it a spin. It has all the cool features we're obviously expecting from a smart watch nowadays: step counter, heart-rate monitor, Bluetooth connection to your cell phone, showing notifications from your phone on your wrist, navigation, voice commands, etc.

Compared to the original Galaxy Gear I tested a year ago, however, the notifications are actually meaningful on the watch now. You get a preview of important emails, text messages, WhatsApp, FB Messenger, and any other app that properly uses the notification API in Android.



If you want to see more of the notification, you just swipe up and get to see the whole message:



Ah, much clearer now. The question was about a college course, not a reptilian issue. The new Android Wear platform also allows you to tap such a notification on your watch, and the corresponding app on your phone gets launched. So if you need to respond to an email, just tap your watch as you take your phone out of your pocket, and you're right where you need to be.

If you need to do a quick Google search, you can now do that from your wrist with voice input and also see the top three results right on your wrist - or tap them to open them on your phone, if you need more details. For example, a search for "XML Editor" produced this:



On the hardware side, the moto 360 gets a lot right, that other smart watches got wrong. The charging cradle provides wireless charging and turns the watch into a nice bedside table alarm clock. There is only one button on the watch, and it is exactly where the crown used to be on mechanical watches. All other user interaction is done via the touch screen with intuitive swipe operations. The wrist strap is available in leather now and a metal version is coming later this year. The watch body is stainless steel with the glass surface being Corning Gorilla glass.

Last, but not least, you obviously get a choice of 6 different built-in watch faces, including a nice retrograde display, and you can download and install additional watch faces from the app store - some of which are nicely customizable.


What I need now is for some clever app developer to create a really beautiful watch face that includes some of the classic complications: equation of time, moon phase, sunrise and sunset times, sidereal time, etc.

Building a stand-alone mobile solution with MobileTogether

In a recent blog post I introduced our new MobileTogether platform for building mobile in-house solutions. Today I would like to give you a little demonstration of how easy it is to build a mobile solution with MobileTogether Designer.

As an example, we're going to build a simple tip calculator app for your next restaurant visit. Since this particular solution doesn't need any back-end data, we're going to create it as a stand-alone mobile solution so that it can be used even without a server connection once it is deployed.



As you can see, it just took about 8 minutes to build this app. MobileTogether lets you focus on what is really important, and handles everything else for you. If you want to try for yourself, you can download MobileTogether Designer here.

You can also watch more MobileTogether Designer video demos here.

Pixelstick

In December last year I contributed to the Kickstarter campaign for Pixelstick - an interesting new photographic tool for light painting. When our Pixelstick arrived in early September, it was immediately clear that it would need to go to New York with my son when he went back to college.

Calvin is a photography student at NYU's Tisch School and so I knew he would put the device to some creative use... and indeed he just posted a "How to" video on YouTube:



For more information on Pixelstick go to www.thepixelstick.com

For more updates on Calvin's work, follow him on Twitter @EpicFalkon or check out his website discover.calvinphoto.pro

XQuery Update Facility in XMLSpy

A really cool new feature in XMLSpy 2015 is the interactive XQuery Update Facility support, which lets you make changes to XML instance documents in a programmatic way - using XQuery statements - that exceed the typical regular expression based Find/Replace capabilities by far. The XQuery Update Facility specification provides a mechanism to insert nodes, delete notes, and modify nodes within instance documents. In XMLSpy 2015 you can now apply these updates either to the current file, all open files, all files in a project, or to entire directories.

This video explains the most important XQuery Update Facility commands and demonstrates how easy it is to put the power of XQuery Update Facility to work for you:



The new XQuery Update Facility support is one of the many new features introduced in the new Altova version 2015 product line last week, which includes new versions of XMLSpy, MapForce, all the other MissionKit tools, and all Altova server products.

Introducing MobileTogether - Build mobile in-house solutions faster

It's about time that I start talking about our next major product here: Altova MobileTogether is an exciting new cross-platform mobile environment that lets you build in-house mobile solutions for your workforce much faster and more productively than any other mobile cross-platform method out there. You can use MobileTogether to bring your in-house data — be it in SQL databases, XML, available as web services, etc. — to your employees on the device of their choice, be it business intelligence dashboards, elegant enterprise forms, or any other business processes from graphs for sales analytics to monitoring of business-critical data.


There are, of course, many ways to develop mobile solutions, and for external customer-facing apps the native platform approach or other multi-platform SDKs may make sense. But for in-house solutions the math just doesn't work. You need to be able to build these in a few days rather than weeks or months in order to stay on budget.


MobileTogether makes this rapid development possible by using a unique system architecture that consists of the following three components:


MobileTogether Designer

MobileTogether Designer is the IDE where you build your mobile solutions. It comes with full database-support for all major database servers as well as the ability to connect to any XML files, web services, HTML pages, or other data sources directly. If that's not enough, you can connect to FlowForce and MapForce server as an interim data transformation platform to get your data from EDI or other formats into XML easily.


Once you've defined your data sources, you then drag & drop UI controls onto your design surface and connect them with the data model. You can build powerful program logic using visual ActionTrees and for data manipulation the full power of XPath and XQuery is at your disposal.

This way you can build a powerful mobile solution in just a few hours and can test it right inside MobileTogether Designer using the built-in simulator that supports the iOS, Android, Windows Phone 8, and Windows 8 look&feel as well as many screen sizes and device options.

We have put together a few video demonstrations that show how easy it is to build a mobile solution with MobileTogether Designer.

MobileTogether Mobile App

Once you are satisfied with the way your mobile solution looks, it is time to get it onto your mobile device. The MobileTogether Mobile App is what runs your solutions on your device, and it is available for free form the respective app stores. The Mobile App is available for iOS, Android, Windows Phone 8, and even desktop Windows 8, so you can deploy your solutions to all mobile workers, no matter if they prefer a smartphone, tablet, or laptop!


Normally the MobileTogether Mobile App connects to a MobileTogether Server (see below), but for an initial trial run, you can simply connect the MobileTogether Mobile App directly to your MobileTogether Designer. Instead of starting the simulator in the designer, you select "Trial Run on Client" from the toolbar, and then you can see your mobile solution on your device and test it there, provided your mobile device is on the same local network as the computer where you are running MobileTogether Designer.

MobileTogether Server

Once you're ready to deploy your solution to your entire workforce, it is time to install and configure MobileTogether Server. This server acts as a conduit between your mobile clients and your database servers and other data sources in your IT infrastructure.


If you only want your mobile solutions to be available while the mobile devices are connected to your corporate Wi-Fi network, then it is sufficient to install the server in-house and your employees can immediately run your mobile solutions, once you've deployed them from the Designer to the Server.

If you want your employees to also be able to access your solutions while you are on the road, you will need to designate and open a port in your firewall so that the client devices can reach your server from the public Internet when they are traveling. We recommend installing an SSL certificate for that purpose so that the data connection between the clients and your MobileTogether Server is encrypted. In addition, we recommend securing the MobileTogether server with user authentication. You can choose between built-in user management, or the MobileTogether server can talk to your Active Directory server to integrate with your enterprise user management.

Alternatively, you can also install MobileTogether Server into a private cloud with a cloud provider of your choice, if you prefer to have your server running in a cloud rather than on your on-premises infrastructure.

Timeline

In May this year we first introduced Altova MobileTogether at TechEd in Houston, TX. In July we launched beta 1 of the MobileTogether Designer. In August we launched the beta 1 version of all the MobileTogether clients in the respective App Stores. And last week we launched beta 2 of MobileTogether Designer, Server, and the Apps. We expect MobileTogether to be commercially available later this fall.

Getting Started

You are invited to participate in the beta 2 and try it for yourself.


Just download the MobileTogether Designer from our website, download the MobileTogether Client from the App Store on your device, and you can be up and running and have your first solution on your device in 1-2 hours. Then, when you want to scale out to have your colleagues run it on their devices, you can download MobileTogether Server so that others can connect to it.

New XBRL Formula Editor in XMLSpy

I'm very excited about all the new features in XMLSpy 2015, and in particular about the new XBRL Formula Editor, which now lets you build formulas more intuitively using a XBRL Table Linkbase layout preview. The XBRL Table Linkbase specification provides a mechanism for taxonomy authors to define a tabular layout of facts. The resulting tables can be used for both presentation and data entry. In XMLSpy 2015 we use these tables as a way to specify variables in XBRL Formula definitions.

However, XBRL Table Linkbase is a fairly young specification, so not many published XBRL taxonomies include Table Linkbase definitions yet. Please see my previous blog post, where I have demonstrated how to add a Table Linkbase to an existing XBRL extension taxonomy.

Once we have a taxonomy with a Table Linkbase attached to it, we can then proceed to create some assertions or calculations using XBRL Formula Editor. In this video, I will demonstrate how to do this using a recent SEC filing as an example instance:



The new XBRL Formula Editor is one of the many new features introduced in the new Altova version 2015 product line last week, which includes new versions of XMLSpy, MapForce, all the other MissionKit tools, and all Altova server products.

XBRL Table Linkbase Editor and Layout Preview

This week we launched our new Altova version 2015 product line, including new versions of XMLSpy, MapForce, all the other MissionKit tools, and all Altova server products.

One of the cool new features in XMLSpy 2015 is the real-time XBRL Table Linkbase layout preview. The XBRL Table Linkbase specification provides a mechanism for taxonomy authors to define a tabular layout of facts. The resulting tables can be used for both presentation and data entry.

However, XBRL Table Linkbase is a fairly young specification, so not many published XBRL taxonomies include Table Linkbase definitions yet. This is where XMLSpy can greatly help: in this video I will give you a quick demonstration of how to add a Table Linkbase to an existing XBRL extension taxonomy, using an XBLR filing that was submitted to the SEC as an example:



Learn how the graphical XBRL Table Linkbase editor in XMLSpy makes it easy to define XBRL tables for the presentation of multi-dimensional XBRL data. You can determine whether your table produces the desired results in the real-time XBRL Table layout preview, which is new starting in XMLSpy 2015.

How to download and process SEC XBRL Data Directly from EDGAR

Earlier this year I presented a webinar for XBRL.US where I demonstrated how you can use the vast number of XBRL filings that have been submitted by public companies to the SEC and are available for free to download from the SEC's EDGAR system:



Since then I've occasionally received requests for the slides used in that webinar, and the slides are available on SlideShare now.

In addition, several people wanted to see and reuse the complete Python scripts that I demonstrated in the webinar, so I have now uploaded those and published them in a new GitHub repository:

https://github.com/altova/sec-xbrl

These scripts are available under an Apache 2 license and require Python 3 as well as RaptorXML+XBRL Server installed on your machine. For more details, please see the README file published on GitHub.

Computer brain surgery or How to performa a RAIDectomy

I have an old 2010 Mac Pro in my home office that is my main photo editing machine (using Adobe Photoshop CC and Lightroom 5). It also serves as a remote desktop terminal to my office PC. It has 2 Intel Xeon CPUs with 6 cores each running at 2.93 GHz and 32GB of RAM, so even by today's standards, 4 years later, it is quite a powerful machine.

At least in theory it should be. Back when I bought the machine I thought it would be a good idea to get the Apple RAID card and 4 drives with 2TB each and set them up in a RAID 5 array for data protection, giving me a total usable 5TB of disk space, which I  set up as a 2TB boot drive and a 3TB drive for data.

Apple RAID Card and Drives
Apple RAID Card and Drives

That RAID card, however has been giving me nothing but trouble in those four years, and it got so bad this summer that it was time for a radical move: RAIDectomy!



The issues that I experienced with the RAID card were the following:

  • Every so often the RAID card would complain that the on-board battery was not fully charged, and would disable the write-cache, resulting in a severe performance hit that slowed down the entire machine to a crawl.
  • Every couple of months the RAID card would enter a mode called "Battery reconditioning", where it also disabled the write-cache for a day just for good fun, and there was no way to stop or postpone that process. If you wanted to get any work done that day, you were out of luck.
  • About every 3-4 months, the RAID card decided that it was time to rebuild the RAID, so it went into a 48 hour mode of scratching all the disks 100% and the computer was unusable during that time.
  • Then the RAID card informed me that the battery was dead and disabled the write-cache permanently last year.
  • Even after replacing the RAID card battery back then, these issues did not go away, but were rather just suppressed for 4-5 months, before they resurfaced.

And now the RAID battery has begun a few weeks ago to give me the impression that it was going to die again soon, so I decided to take a more radical approach this time and get rid of the RAID card once and for all.

I also decided to replace the hard drives and get something with a bit more speed and less wear and tear to reduce the risk of data loss, so I bought a Crucial 1TB SSD drive and a new Seagate 3TB 7200rpm hard drive both with SATA interfaces. I figured I would use the SSD as my boot and application disk, and the larger 3TB hard drive for photo storage. To ensure that the smaller 2.5" SSD drive would properly fit in the 3.5" bay, I also purchased a conversion bracket.

So the process I had in mind was to copy the data over from my two old RAID logical drives to the two new drives and then remove the four old drives and the RAID card.

Easier said than done...

Obviously the first step was to do an extra backup (in addition to the TimeMachine/TimeCapsule network backup that was always running). So I connected my 4TB external USB 3.0 hard drive and started the copying process - only to realize that Apple in their infinite wisdom only equipped the Mac Pro with USB 2.0 ports and the only fast external port on that machine was a FireWire 800 connection. Of course, nobody still uses FireWire on this planet and no external hard drive in my collection supported it, so I had to wait 9½ hours for 3 TB of data to copy over to that external drive using USB 2.0.

This also meant, that my original plan to restore the data onto the new disks from the external drive was going to be more painful than I was willing to entertain.

So I decided to do a 2-step approach. The Mac Pro luckily has two 5.25" bays for optical drives and I had only one of them filled, so I had a SATA connector available on the inside of the chassis that was not controlled by the RAID card. This allowed me to connect the new 1TB SSD inside of that bay and just let it sit there without any screws attached. Upon powering up the machine, I used Apple's Disk Utility to format and partition the new drive, and then planned to use the "Restore" function inside that software to clone all the data and the recovery partition from my old boot volume on the RAID onto the new SSD boot disk. Indeed, there were many online support discussions that I found that praised the ease of using the Apple Disk Utility for that process.

What all these discussions and help forum posts failed to mention is that you cannot use that process to clone the active startup disk on a Mac. There is simply no way to do it with the built-in tools.

After a bit more research I found a nifty utility called Carbon Copy Cloner that promised to do exactly what I needed, and it offered a free 30-day trial, so I downloaded it and gave it a spin. Indeed, it was not only able to properly copy my entire boot volume from the RAID to the SSD, it also correctly copied and built the recovery partition for MacOS X. Huge tip of the hat to this software, and after I saw it working so flawlessly, I did, of course, purchase a license.

After the boot disk was cloned, the next step was to repeat the same process with the data drive: I disconnected the SSD, put the new 3TB drive into the 5.25" bay, used Carbon Copy Cloner to copy all the data over, and then removed it again.

Now it was time to perform the actual RAIDectomy and remove the 4 original drives and extract the RAID card. That process went very smoothly, and I was also able to quickly mount the 2 new drives in the main drive brackets and insert them so they connected directly with the backplane.

I was pleased to see the machine boot properly from my new drive, and even more excited to see the vastly improved speed of everything. This four year old Mac feels like a new machine now.

Since there was no good articles online on how to remove an old Apple RAID card, I figured I'd share my experience here - in case anybody else out there is contemplating getting rid of their RAID card.

Obviously, the entire process would have been much smoother, had Apple actually supported USB 3.0 in that machine rather than FireWire 800 as the only high-speed external port. Even more important, a product like the Apple RAID card should never have been sold in the first place. It was poorly designed, suffered from battery issues, and slowed down the machine at random times outside of the user's control.

The (Broken) Promise Of Wearables

This week the Moto 360 became available and sold out in record time. Next week Apple is rumored to introduce their iWatch along with the next generation iPhone. And Samsung just announced their next generation Gear watch last week that is expected to ship at the same time as the Galaxy Note Edge.

It is clear that Wearables hold a lot of excitement and a lot of promise, and obviously capture people’s imagination. But let’s pause for a moment first, and talk about past experiences with Wearables:

Wearables on a table

Among the many tech gadgets that I’ve acquired in recent years, I also purchased the first Samsung Gear watch when it came out last year, I bought the Google Glass Explorer Edition, and I bought the Fitbit fitness tracker.

Each device held a unique promise that it would make life easier and would add useful functionality. And each device ended up breaking that promise in subtly different ways.

My experiment with the Galaxy Gear smartwatch was very short-lived. The essential promise was to show important information about incoming emails or messages without the need to have to reach into your pocket and get out your phone. However, it wasn’t ubiquitous. Apps on your phone had to be written to be compatible with the watch. And for the most important communications medium – email – all the watch would tell me is that a message had arrived from somebody. No subject line. No relevant content summary. It was essentially useless.

The Fitbit fitness tracker held a simpler promise: wear it all day and all night and track your health. Except that it is so small that I lost it about 3-4 times during travels and only rediscovered it again when I emptied the suitcase after the end of the trip. Here it was the need to go to your computer and sync it there that broke the promise of ease of use. And the tracking information was primitive at best – a simple step counter. I gained no useful benefit from the device that would really help with my weight loss efforts.

Google Glass was perhaps the biggest disappointment of the three. The promise of having an augmented reality experience sounded great. But all it really did was provide notifications about messages, news, and occasional information from other Glass apps about restaurants, sightseeing spots, and other trivia. And after a short while any person wearing Google Glass was labeled a Glasshole and privacy concerns soon resulted in sings being posted everywhere that Glass was not welcome.

In the end, all three devices suffered from a lack of really useful and unique features that actually provided a real-world benefit combined with the need to be constantly charged using various different charging devices and having a fairly short battery life. The day came when I forgot to charge them and then left them at home. Then the day came when I forgot where I put the charger. And now they’re all sitting in a drawer somewhere…

So I look towards the new generation of Wearables coming out this fall with a bit of cynical skepticism: how long until they, too, end up in a drawer somewhere?

In the end, there are only two devices that I carry with me all the time and every day:

  • My watch is a Seiko Astron Kintaro Hattori Limited Edition. It is a solar-powered watch that has a built-in GPS receiver and synchronizes time with the atomic clocks aboard the GPS satellites. And it also automatically adjusts your time-zone when you have landed – based on the GPS position. It never needs to be charged and it does one thing extremely well that I care about: tell accurate time.
  • My phone is a Samsung Galaxy Note 3 (soon to be replaced by the Note Edge). I’ve gone through many smartphones over the years, and it is my favorite so far. I hate its battery life. I hate that I have to charge it. But it does so many things so well in just one device, that it’s worth the hassle. And I use it all the time.

That last point is, perhaps, the biggest question about Wearables that we should ask ourselves: if we already have our phones in our hands all day anyway, what is the additional benefit of a wearable device?