System Alert! Most Systems Down

Update 10:20 am Pacific.  Systems  restored and functioning normally as of approximately 9:30 AM Pacific  Testing continuing but apps and services appear to be working properly

———

A critical hardware failure at one of our system providers is causing all FlyQ, eCommerce, and most other systems to be offline.  The provider has been looking at the failure and working to resolve it.  At the moment, we do not have a definitive ETA of system restoration.

We’ll keep you informed as we learn more and deeply regret the problem.

29 thoughts on “System Alert! Most Systems Down

  1. Major fail on Seattle Avoonics part that the app has to talk to a system. the system is down and this app is now useless because it thinks it has to login. I have a lifetime subscription and other than failing to get updates I should be able to use the app but no. Login screen come up and makes it utterly useless. I am speechless at this FAIL in the design!

    Like

    • Hi Stig, quite right. We have multiple redundant servers for every part of the system except one (the main database) that has been running flawlessly for 14 years. We were already in the process of migrating it to a cluster of database servers. Now we’re doing it faster. Very sorry for the issue and we’re working to make sure it never, ever happens again.

      Like

  2. Thank Kieth Russo for the lighting quick response on my email regarding the outage. I have total confidence your service will only improve because of this.

    May I suggest a programming upgrade that would allow the offline map function even when the server is down, instead of the login loop.

    Cheers

    Like

  3. Yes, it may suck but come on, life in not always perfect. Give Seattle Avionics an little grace on this. I’ve been with them from the very beginning and this is their first glitch. That’s a good record in my book. Hopefully they will have it fixed soon.

    Like

    • Have to disagree,

      Not sure what your definition of a “real pilot” is.

      I fly Angel Flight missions in almost all kinds of weather except level 2 and above, typically IMC for 80%. Sometimes short duration in Level 2 — using a C-172.

      I utilize FlyQefb from either my desktop or tablet to check weather, NOTAMs, TFRs, make last minute adjustments to IFR flight plans, file with the FAA, and save/print my documentation for the mission.

      The ability to use one software application to perform these tasks is more rapid and accurate when done just prior to departure to the airport. Speed and accuracy are but two of the components for a safe flight for my patient, aircraft, and me.

      When FlyQefb is not operating, the stress seems to always lead to errors in the flight planning as well as incomplete documentation for the mission.

      Yes, I have practiced for outages, and I am not perfect when my departure time will be delayed on account of hardware/software issues. I do not need the pressure.

      Don’t know where you fly. I fly from central Florida both north towards TN and south.to the Keys. Almost 500 miles for each mission leg. And I have frequent flyers who I transport on a regular basis.

      This to me is a “real pilot.” The instructors I have met typically have less IMC experience than I have. including flying in/out of Class B airports in the Southeast.

      I think I qualify as a real pilot.

      Like

      • “When FlyQefb is not operating, the stress seems to always lead to errors in the flight planning as well as incomplete documentation for the mission.”

        This indicates you are relying too heavily on this particular tool. This is an aid to flight planning and navigation. You should consider becoming proficient again without the use of FlyQ or any other EFB before carrying passengers again.

        Like

      • Jason, I understand where you are coming from. I am quite proficient in using other methods.

        To summarize, the data I have in the cockpit needs to be readily available and up-to-date. This means that if one cannot log into FlyQefb, one of my real-time tools becomes no longer available.

        I am not sure you actually comprehend MY requirement for a workflow which rapidly produces the desired end result.

        My flights typically require an early morning departure from my home to final prep the aircraft. This means I need a good night’s sleep. This also implies that I need up-to-date flight planning info — which will probably be stale when I take off (I live and fly out of Central Florida). Before leaving for the airport, I typically adjust my three flight leg plans (to reflect weather and other updates before I file them
        .
        In my real world as a private pilot, almost every flight I plan requires revamping in real time due to weather changes, traffic, and pop-up TFRs. These flight legs are over over two hours long, and ATC typically amends most plans during the flight — after I have taken off (all this while I am flying single pilot IFR,.

        In my IFR world, I work with SIDs and STARs on every mission, which have various entry and exit options. Yes, I can alter these manually from a procedure page, but that takes time and distracts from the real time requirement to fly the aircraft, especially in moderate turbulence.

        Likewise, casual pilots are not flying in the locales I fly or at these hours. The only other aircraft I hear are package and passenger carriers.

        For example, while flying back yesterday from Marathon (Florida) to Orlando Sanford (Florida), at sundown, ATC assigned me RSW as the entry to the MINEE Five arrival; I originally filed MOANS as the entry.

        ATCs assignment of RSW had me flying directly into a storm with heavy precipitation and lightning; ATC (Miami Center) on my request did allow me to deviate 50 miles to the west into the Gulf, but refused to allow me to fly east 20 miles to completely avoid the system. Reason #1: ATC’s display (as well as mine), showed the storm in a location to the east where it did not exist. From experience I have learned the weather display could be anywhere from 15 to 30 minutes old.

        I made calculations before departure and added 10 gallons of fuel, which gave me about five hours of endurance for a two and one half hour flight. Rather than declare an emergency, I decided to test whether ATC (Miami Center) was wrong and I accepted my assigned routing.

        Because the first storm sat over RSW, ATC (Fort Myers) later assigned me to fly direct to MOANS — my original plan as filed!

        Reason #2: Hmm, I could have easily done this at the outset, except I would have overflown an active MOA which began at 7,000. While I had departed Marathon at 3,000 and was later assigned to 8,000, I considered amending the plan in flight to descend to 6,000 or 4,000 and fly under the storm system.and avoid the MOA, except the wind forecast at these altitudes was brutal and there were no PIREPs..

        Like

  4. Steve/FlyQ Support: I think you’ve done a great job of maintaining access to Fly Q, and I was impressed by how quickly you got it back up and running. One observation and question: I tried opening Fly Q EFB on my iPad Pro and it hung up at the password window; It left me unable to access even the recently updated maps. I have a back-up iPad 2, and on that, before opening Fly Q EFB, I first turned off the cellular and WiFi connections, got no password request, and was still able to access the maps. So, should you give us some downtime procedures in the case of a future downtime, and is it further possible that we should use airplane mode or turn off cellular/WiFi to preserve access to the info already downloaded to our devices?

    Like

    • Hi Bill, in general, the system works if it can’t connect to any server (like in flight). With the database down, the servers were responding No Subscription. Thus, the machine that was not connected to the Internet worked fine.

      Like

      • Did you consider taking the server down after the database went down? This would have prevented the catastrophic impact to all users attempting to connect and get updates/weather before a flight.

        Like

      • Yes but it would have caused problems including for our other customers who use Dynon, Bendix-King, Aspen, etc. systems. Also, because of the exact timing of the failure, we had our key IT folks about to board a commercial plane for OSH so if any issues surfaced from taking down the servers, 100,000 people may have had a problem for 5 or 6 hours.

        BTW – FlyQ is NOT designed to fail if the servers go down — rather the opposite. What happened that day was that a 3rd party database went down, something that had never happened in 15 years of using that system. Due to a bug in the way the server system (which was still running) handled that unique problem and a small bug in EFB itself, the app got confused and requested a login since it thought it got a reply from the server saying your account did not exist (which, with the DB down, was true to a literal system like a computer).

        We have tested and deployed a fix to the server that helps there and the next release of FlyQ EFB (in final beta testing now) had further guards against it. We have also started to migrate the database from the current provider to a much larger provider that operates much more redundant systems.

        Thanks, Steve

        Like

  5. I’m sure you have this in mind now but thinking that you can find a single 100% uptime platform is erroneous. If you are going to be a serious service, you need redundant fail safe system with dispersed mirroring.

    Like

    • Hi Rob, quite right. We have multiple redundant servers for every part of the system except one (the main database) that has been running flawlessly for 14 years. We were already in the process of migrating it to a cluster of database servers. Now we’re doing it faster.

      Like

  6. You should always have a backup plan or procedure for obtaining information for your flight, for those situations when your preferred method is on the blink. That’s a real pilot.

    Like

    • Hi Mike, quite right. We have multiple redundant servers for every part of the system except one (the main database) that has been running flawlessly for 14 years. We were already in the process of migrating it to a cluster of database servers. Now we’re doing it faster.

      Like

  7. I’m sure you are already thinking along these lines. Just in case, the kind of 100% up-time that you need is not possible with a single service platform and/or provider, whatever they may tell you. You need redundancy and wide area mirroring. To achieve this kind of reliability, we go with two providers that deploy different architecture and are mirrored widely.

    Like

    • Hi Rob, we have multiple redundant systems for every part of the system except the main database. It worked perfectly for 14 years and we were already migrating it to a cluster of redundant DB servers so this accelerated that.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s