ReleaseNotes.txt

# See ReleaseNotes.txt for detailed version history
#
# V1.41 2024-Feb-04
#   Put Season / Episode providers in pigs.json so dont have to recode if more providers added
#   "SEP_List": ["Gracenote","TVmaze"]  List searched in order, can extend list if needed
#   modified ls.py to use SEP_List
#
# V1.40 2024-Jan-12
#  sv.py Typo /mycons
#  sd.py Some cleanup
#  cm.py Change CM server message to .debug - URL confusing when seen in the logs
#  pigsdns.py validate IP addresses in PigsDns 'hosts', with log messages to help
#  Validation of config Server_IP and -s cmd line with help messages
#  Change pigscfg[] to Settings[] in sv.py and ls.py and PigsSettings[] in pigs.py
#  find.htm and se.py - add a checkbox for [x] Start search 'Now' (start near current time)
#  Add timestamp to /pigstatus and save in a file pigstatus.json on every view for debug
#  Updated format of /pigstatus. Move boring stuff to bottom
#  Add UI checkbox UseMyIcons, allows user to disable 'myicons' to check for new SD icons
#    You can always see what Icons are in use with the Lineups(view details) line on the UI
#    when [ ] Use MyIcons is not checked, icons in myicons are bypassed
#  Added file pigsreq.txt, a pip 'requirements' file to install Pigs dependancies
#    After unzipping PiGS files to a folder, open a terminal or cmd window and type
#     sudo pip3 install -r pigsreq.txt   (if pip3 doesnt work, just 'pip' might)
#    This will install/verify python-dateutil, pillow(PIL), and requests in 1 step
#  Added Required modules test and install instructions in case modules are not installed.
#  Added Update Status "Barker" to Channel guide.
#    PigSettings['BarkerChan'] = "3.4"     use = "" to disable
#    Every listing Title on that channel  will contain last
#        DVR Guide Update time like: "Updated Tue 02-13 11:13"
#    Every Listing description will contain last SD Download & Dvr guide update:
#     "Schedule downloaded from SD on Tue 02-13 08:13. This DVR Last Updated Tue 02-13 11:15."
#    I put mine on my QVC channel, which happens to be my 4th channel, so its easy to see
#    Note: SD build time wont (cant) show until DVR does its daily update
#  Improved Startup
#    Service list now Saved and Loaded (i think that is now whole PiGS state)
#    PiGS can now serve DVRs w/o contacting SD when it (re)boots using saved info
#    DNS is turned on immediately if schedule was loaded from disk
#    New command line options -holdoff & -delay <Hrs> can be used
#      to delay the build and/or holdoff DNS (allows sync of DVRs also)
#  Added command line option -delay -d <hours> to Set initial build delay to
#    allow user to set first build for tomorrow morning for example.
#    Also prevents 1st dvr request after boot from resetting build time
#  Added command line option -holdoff(-o) to hold off DNS (thereby DVRs)
#    until after first build occurs. I like my DVRs to update sometime
#    in the morning, but they slip forward over time.
#    I can quit PiGS then restart with
#      sudo python3 pigs.py -holdoff -delay 30 (30hr = 1 day + 6 hrs)
#    This starts PiGS, keeps DNS disabled and schedules the first build
#      in 30 hours (set hours to when you want build to occur)
#    The delay is set to be anytime after your last DVR is expected to update.
#  Added command line option -update -u
#    updates pigs.json with any new configuration keys and exits.  The
#    update happens regardless, this just exits so you can edit immediately.
#  Added Profiling Timers to build for /schedule /programs to see SD timeouts
#  ls.py 1-12-24 Fixed Season/Episode numbers after TVmaze data added 
#  pigs.py fix SaveSettings with -u option, remove Barker default 3.4
#  pigs.service Added KillSignal=SIGINT so PiGS exits/restarts gracefully on 
#    sudo systemctl stop pigs or restart pigs
#  Added dirty flag to UpgradeConfigFile(), user settings now saved if updated
#
# V1.39 2023-Jan-04
#  Remove import auto() from enum... its not Python 3.5 compliant and I dont need it.
#  ls.py fix missing Prog detail error description - used pidstr instead of apidstr
#  Add retries on SD GET requests.  Helps build succeed when SD server under heavy load.
#  The Acct API's (PUT/DEL's) do not have retries so if the SD server sends 500 errors
#    a user could skip account verification to maybe get past startup/init
#  Added config key and cmd line -noverify -v to skip SD account verify step on init
#    "VerifyLineups": True = Verify Lineups, False=skip verification
#  Add SD.py msg 'Retry succeeded.  Continuing..."
#  Set build status when loading Sched from disk
#  Added 'CName' functionality to PiGS DNS server for NTP pool redirection
#    e.g. in pigs.json, redirect default pool.ntp.org to server farm of your choice
#   'pool.ntp.org' : 'time.nist.gov',     # redirect time server requests
#  Fix build in LoadProgramsFromDisk - tried to access non-existant schedule
#  Add try/except to utGetPigsVersion() float() conversion in case bad valve
#  Add valid json to GetToken return value to prevent an exception
#  Increase the SD Login timeout from 10s to 90s since SDxx now does retries
#  Added msg "Login Status : Trying..." to UI during login attempt
#  Additional error checking on POST command - validate json data &
#    reject unsupported POST comands (9/21/22)
#  Add GET /linearschedule parameter checking if DVR makes bad startdate
#     requests and return HTTP 400 for bad data (12/28/22)
#  Added dPiGS['SleepMonitor'] key and code for Installs where host PC may sleep
#    Python timers loose time while PC sleeps. Now monitor the expected build
#    Real time and start a build if the time passes (like when the PC wakes up)
#
#  Improved Schedule Remapping support.  DVR+ doesnt like it if it has 
#   duplicate stationID's and will only display one of the schedules.
#   It would show up re-mapped, but maybe gone from previous channel.
#   To get around this, I added a 'schedule' key where you put the
#   CallSign of the stations schedule you want to display.  I use this
#   info to look up the real schedule and this provide a unique StationID
#   for each station so DVR will display the same schedule on both chans
#   You can map one stations schedule to another by adding 'schedule' key.
#   since its json, you can even add a 'comment' key if you wish.
#    {
#      "channel": "15.2",
#      "name": "WMTVDT2 (WMTV-DT2)",
#      "callsign": "WMTVDT2",
#      "affiliate": "CW",
#      "schedule": "WISCDT",
#      "comment": "put channel 3.1 WISCDT schedule on channel 15.2",
#      "favorite": true
#     },
#
#  Added psuedo CallSign MYICON which allows you create a 'fake'
#    station to add icons to stations that do not have SD schedules,
#    or if you want an icon, but want the PSIP OTA schedule.
#    Place the icon file you want in the folder /myicons e.g. /myicons/3.4.png
#    {
#      "channel": "3.4",
#      "name": "WISCDT4 (WISC-DT4)",
#      "realcallsign": "WISCDT4",     <<<<<< changed key of old sign
#      "callsign": "MYICON",        <<<<<<< 
#      "affiliate": "QVC",
#      "favorite": true
#    },
#
# V1.38 2022-May-14
#  Program Schedule and Cache Saved to disk on good build in xxx.pkl
#    Needed way to keep Cache between runs while making code changes 
#    Also ensures a Schedule if there's an SD outage on PiGS boot
#    Added key "SaveSchedule" : 2,
#       0=Dont save, 1=Save Cache, 2=Save Both Schedule & Cache
#    Saved files are loaded if they exist.  Can be manually deleted w/o harm
#    Cmd line flag -n -noload prevents Both from loading
#  Make sure md5 Stale Cache items are in ToSaveSet() so Cache current
#  DNS was left disabled after a failed build (during 20min retry) even
#    if a sched was available (yesterday or loaded)
#    Now DNS is enabled after a build if a schedule is available
#  Made Test_Webserver work when port<>80, allows multiple PiGS instances (for dev)
#  Build days displays (-1/+17 days) which means 1 prior day, 17 fwd counting today
#
# V1.37 2022-May-05
#  Change 'Removing stale program[]...' Cache message to .debug, then display a final count of removed
#  More build logging, Do SD Post retries on SD 50x errors as they can be quite transient
#  Remove Unused 'Chanfilter' key
#  Yikes, in 1.35/1.36 I had a default value of '07:00' in FixedBuildTime, so NEW installs will build every day at 7am
#    so in 1.37 I remove FixedBuildTime and Change to StaticBuildTime to fix those few installs
#  Create UpgradeConfigFile() near Config keys to group config update near definition
#  Add "LogFileBytes" : 250000, "LogFileCount" : 3   to config file
#
# V1.36 2022-May-01
#  Add time a Cache item was last accessed & purge old items, otherwise it could grow indefinitely
#  Add some build.info logging
#  Test PiGS version from pigs.goihl.com/version.txt, notify user if update needed
#  index.html, pop up message if Update needed
#
# V1.35 2022-Apr-28
#  Remove LastSchdBuild & LastSvcBuild keys from config dictionary.
#  Chg timer.SetName(), .SetDaemon() due to deprication
#  remove geteuid() test for root, it only works on Linux.
#
# V1.34 2022-Apr-24
#  Fix boundary condition of ls.py when everything is in cache, nothing to look up at SD
#
# V1.33 2022-Apr-22
#  Change logging of SD Api
#  Change Webserver Testing to a HTTP HEAD with a timeout of 10 seconds
#  Use an event to sync WS Reset to Builder so messages are not interleaved
#
# V1.32 2022-Apr-21
#  Bug in Get token when no UN/PW, Tie Program Cache Clear to Build mode
#
# V1.31 2022-May-xx
#  Fix SD.py GetToken bug when UN & PW are blank.
#
# V1.30 2022-Apr-20
#  SD's is(will be) restricting repeat Program ID's requests so duplicate PID's need to be Cached locally
#    ls.py BuildLinearSchedule() restructured to use Python Sets and a small memory Cache
#  Changed SD http API requests to HTTPS because thats what SD wants
#
#  PiGS requires Python 3.7 to use the threading http server
#    Added a hard exit and Help Message if Python 3.7 is not installed
#  Franks Pi zero was having issues with 1.24
#    sd.py Increased connection timeout and added 5 retries to SD schedule and programs API's
#  1.24's MarkAsNew code did not work... I broke it with too much optimization.  Now fixed.
#  Add Fixed build time - eg Build at 7:23am ever day  
#    PiGS can build at a fixed time each day by setting "FixedBuildTime": "07:00"in pigs.json
#    This is not recommended since a DVR could be in the middle of an update and pigs would just
#    disable dns and do its build possibly causing dvr to timeout with incomplete schedule update
#  Added key 'ThreadedHTTP' to choose normal or threaded http server.  I noticed some missing sched
#    data when multiple dvrs requesting concurrently.  appeared that two sched searches were
#    occurring at same time, maybe confilicting?  ls.py or http handler may not be thread safe???
#    So reversing the 1.24 threadedHttp in lieu testing port 80 and resetting server if needed.
#   https://bugs.python.org/issue31639
#   I can simulate this problem using the Python interactive GUI
#     >>> import socket
#     >>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#     >>> s.connect(('192.168.1.200', 80))
#   If I now go to a browser and try 192.168.1.200/pigstatus it hangs
#   As soon as I execute :
#     >>> s.close() 
#   the webpage immediately loads. 
#   the blocking seems to happen in http.server
#   https://github.com/python/cpython/blob/3.10/Lib/http/server.py
#       handle_one_request(self):
#           self.raw_requestline = self.rfile.readline(65537)
#
# V1.24 2022-Jan-01 (only released to beta testers)
#  PiGS requires Python 3.7 to use the threading http server
#  sd.py Fixed return value bugs that caused a build exception on an SD timeout
#  sd.py increased SD read timeout to 90s from 30s to stop GetProgramDetails timeouts
#  sd.py Removed F-string formatting so pigs will continue to work in python 3.5
#  Fixed build state bug that prevented 20-minute retry timer to be set
#  My 1.24 PiGS would sometimes stop responding to Port 80 UI/DVR requests.  Wireshark showed that
#    chrome on my Andriod phone was making 2 connections to port 80, 1 for for the webpage(data), 
#    and 1 for the favicon.ico, but sometimes chrome would not use the port and left it open.  
#  To fix this I :
#    1) Changed to ThreadingHTTPServer((hostIP, port), Handler) because docs state: 
        # This class is identical to HTTPServer but uses threads to handle requests by using
        # the ThreadingMixIn. This is useful to handle web browsers pre-opening sockets, 
        # on which HTTPServer would wait indefinitely
#    2) At each build I test that PiGS Webserver on port 80 is responding and shutdown/restart webserver if not.
#
#  PiGS Schedule is only replaced(updated) after a successful SD build
#   (Yesterdays schedule build is not deleted until todays build is sucessful)
#   SD typically provides 17-days of data so if todays build fails PiGS still has sched data
#   This will be useful for always-on installs. Can be disabled with "BuildModeDelete" : True
#
#  Added SMTP Email Alerting for build complete and build error
#  "AlertEmails" : "",    # "Name1<email1@yahoo.com>, Name2<email2@gmail.com"
#  "SmtpUser"    : "",    # gmail address  "someaddr@gmail.com"
#  "SmtpPass"    : "",    # gmail password, "password"  
#     google account settings must allow less secure apps to send emails. make a new account just for pigs?
#     https://myaccount.google.com/security
#     https://myaccount.google.com/lesssecureapps
#
#  scan.py dynamically finds CM cname Alt amzn server
#  Add DNS packet validation (so microsoft portqry.exe works)
#     Just did some basic DNS packet sanity checking so port scanners dont kill PiGS DNS server.
#     portqry was sending a UDP string like "From Microsoft PortQry" which killed the DNS server.
#     portqry -n 192.168.1.200 -p UDP -e 53
#     portqry -n 192.168.1.200 -p TCP -e 80
#
#  Added restart commands to pigs.service in case Pigs dies or fails to start (Linux systemd)
#    Restart=always​
#    RestartSec=30
#
#  update /status with more build info, consolidate some fields
#     add SchdStart": "2021-01-31T23:30:00Z",
#     add SchedEnds": "2021-01-31T23:30:00Z",
#
#  Added 'UpdateIconCache' flag to rebuild icon Caches /sdicons and /cmicons when IconType Menu changes
#  This allows user to 'force' and icon cache update by changing icon type
#  Also added this to UI.  Needed when stations move channels around
#
#  Added 'NetworkWait' = 60  key to extend wait time for an Inital IP address.  user can make longer.
#      also ensure pigs has an IP before starting DNS since redirect wont work without an IP
#
#  Added 'AutoSaveConfig' : 2,  0 will disable auto-save
#      Now with some pigs installs running for 200+ days, last dvr update was not saved in Pigs.json.
#      I should really make a separate 'pigsuser.json' file and make pigs.json read-only
#      pigs.json file is auto-saved every N builds to preserve DVR last update states. typically every 2 days.
#
#  Added 'MarkAsNew[]' list in config file.
#     Allows shows matching certain criteria to be marked as New for recording 'New shows only'
#      I should really make a separate pigsuser.json file and make pigs.json read-only
#
#  Example:
#     my local PBS station is not marking the new Season 21 of "America's Test Kitchen" as New when they air
#     I can use this feature to mark all 'S21' episodes as new for recording purposes
#     I may have to delete some S21 duplicates, but at least this filters out old Re-runs I dont want.
#     It does an 'ignore case' substring match of the title and description you would see on Guide.
#     No UI for this.  stop pigs, add your shows to pigs.json, save and restart pigs.
#     be careful editing the json file.  format has to be exact, dont miss any comma's or quotes!!
#     Can also be used to record only certain New shows, e.g. your favorite NFL team(s)
#       The last T/F parameter sets New shows to False (to prevent recording all NFL games)
#       then marks NFL games matching the criteria to 'New'  e.g., to record the best NFL team:
#         [ "NFL Football", "Packers", True],  <<< The best NFL team even if QB has broken pinky toe & covid.
#       Then on your DVR set it to Record New NFL games on the major networks and it will be recorded.
#
#  "MarkAsNew": [
#         [ "America's Test", "S21", False],   
#         [ "NFL Football", "Packers", True],   
#         [ "Show2", "Season or other Copy to match", False],
#         [ "more",  "Add more lines here if you want", False],
#         [ "Show3", "No Comma after Last item in list", False]
#      ],
#
#
# V1.23 2021-Jan-7
#   Make find.htm work with international chars (UTF-8)
#   Test that schedule exists before trying to search
#   Increase DNS timeouts to 20s and 25s and change GHBN to .debug
#   Add SD connection test to the Init/Startup procedure
#   Increased delay of first schedule build to allow platform to stabilize on system reboot.
#     Pigs could get IP on Synology NAS, but Internet was not ready in 2 seconds.
#     Added config key "FirstBuildDelay": 30,  # 1st build occurs 30s after start 
#     Defaults to 2s for updrades so as not to mess with existing installs
#
# V1.22 2020-Dec-15" Released to Beta Testers for evaluation
#   General
#     Added Simple Search UI <your-pigs-ip>/find.htm  Link on config UI page bottom of Icons box
#     Default output to local time with "Tue 12/05 13:30"
#     Dynamically support new CM's CNAME server "dvrnew-elb-579886328.us-west-2.elb.amazonaws.com"
#     Added http Request Timeouts to SD and CM servers.  Without them PiGS could theoretically hang.
#   Improved Schedule Build
#     When schedule build is unsuccessful(due to SD or DNS err?) PiGS now retries the build every 20 minutes.
#     The PiGS DNS server is disabled dring Schedule Build since there is no Schedule to send to the DVR's anyway.
#     The prevents the DVR from giving up on todays update - instead dVR will keep doing dns requests until PiGS is Ready.
#     When PiGS succeeds in the build, DNS will be re-enabled and DVR should immediately get its update.
#
#   Added Python command line support.  python3 pigs.py -h for help
#     PiGS now defaults to Port 80.  can still do pigs.py 8000 to over-ride 
#     This will allow Windows Portable to start with directly with pigs.exe
#     "Server_Port" has been removed.  Use command line or "IDE_port" in pigs.json for 8000,
#   Tried HTTP/1.1 for chunk testing.  HTTP/1.1 requires Content-Length header
#     also needed to over-ride end_headers() to send a 'Connection:Close' header to circumvent default keep-alive
#     UI Setting was not sending Content-Length (worked without it in HTTP/1.0)
#     The Pi-Zero was not working well with HTTP/1.1 and keep-alive - disabled for now
#   Learned that if DVR's GET request doesn't get a response in 30s, DVR closes the connection, which
#     if PiGS is too slow (Pi Zero?), results in a 'Broken Pipe' error.  Maybe PiGS should drop response at 28s?
#     The Broken Pipe Exception is now caught in a try/except and handled gracefully (or at least better)
#   DNS -
#     Separated send / receive error checking and log for better error identification.
#     Log when DNS requests take longer than 20s
#
#   Improved Initialization.
#     There is now an 'InitState' to ensure PiGS is all ready.
#     There are many reasons that a build could fail.  no pwd, no internet, external dns fail, etc
#     Init State is checked each time a schedule build is requested and completed if needed.
#     You will see these in the logs.
#        InitState.NOT_READY   = 0    # Not Initialized
#        InitState.CM_ALT_URL  =      # Acquired CM Alternate Url
#        InitState.SD_LOGIN    =      # Verify SD GetToken works succeeds
#        InitState.SD_LINEUPS  =      # Verify PIGS matches SD Lineups
#        InitState.SERVICE_RDY =      # DVR Servie List Built
#        InitState.SCHEDL_RDY  =      # Schedule has been Built, ready for requests
#
# V1.21 2020-Nov-17
#    DVR states are saved in pigs.json when pigs quits so last update is available on restart
#    /pigstatus does not show dvrs older than 'DisplayDVR' = 7 days old.  0 means show all dvrs
#    Added new=(false|true) to /search so you can search for just 'New' shows
#    1.20 not released b/c /lineup conflicted with /lineupPreview.  Changed to /showlineup
#
# V1.20 2020-Nov-12
#    Bump to 1.20 for new search function
#    Add traceback exc_info=True to Builder Exceptions to help track down occasional crash
#    sd.py 'classify' SDGetStatus() result and use instead of GetLineups() for AddDeleteLineups
#    if a schedule build exception occurs, clear the Airing list so a parital schedule is not sent. LSClearSchedule
#    On ctrl-C shutdown Builder and DNS threads nicely before exiting. Sched build WAS running after ctrl-C
#    Renamed pc.py to scan.py and simplified command line.  It no longer needs the port=0
#    Usage:   python3 scan.py <zipcode> [ROVI | PIGS]
#     Test CM/Rovi servers      python3 scan.py 53593     (defaults to ROVI)
#     Test CM/Rovi servers      python3 scan.py 53593 ROVI
#     Test PiGS data            python3 scan.py 53593 PIGS
#    To test PiGS, you may need to add "192.168.1.200   epg.channelmastertv.com" to /etc/hosts (using your PiGS IP)
#
#    Added stb.channelmastertv.com": "0.0.0.0",   to DnsHosts since stb.xxx went dead on Nov 11, 2020
#
#    Change DNS GHBN address fail message to .debug instead of .warning
#    Better DNS error messages.  Enhance pigstatus DNS display
#      "PiGS DNS Service": {
#      "Status": "Enabled",
#      "DvrLastSeen": "12:47:24",   Will be 'Never' or a local time
#      "LastErr": "None."           'No DVR DNS request for 300 seconds...' means DVR not talking to PiGS DNS
#    'SaveJson' :true flag saves to DvrLineup.json instead of servicedetailsmsg.json
#    Changed DNS server to only send back 1 IP using gethostbyname() instead of gethostbyname_ex()
#      gethostbyname_ex() seemed to make two DNS requests 'to the net' instead of just 1
#    Change DNS error response to dvr to 'server error' 2, instead of 'bad name' error type 3 on ghbn() exception
#    Reformatting of /pigstatus
#    Fix localtime conversion and printing for standard time change-over
#    Add se.py Provides /search and /lineups
#
# http://192.168.1.200/showlineup will display your channels,RF,image link (for debugging)
#  VCh  RF  CallSgn Aff StaID   Lineup          Icon Link
#   3.1 11  WISCDT  CBS 25124   USA-OTA-53593   http://epg.channelmastertv.com/sdicons/3.1.png
#   3.2 11  WISCDT2 MNT 25125   USA-OTA-53593   http://epg.channelmastertv.com/pinkpig.png
#
# for url searching of PiGS Schedule DB (used for DST testing).  See release notes
# http://192.168.1.200/search?expand=&duration=&startdate=&zulu=&channel=&title=&copy=
#     title=&          # string to find in Title (case insensistive)
#     copy=&           # string to find in description Copy (case insensistive)
#     channel=&        # only display listings for specific channel 3.1
#     startdate=&      # Zulu starting time format must be exact  2020-10-31T00:00:00Z
#     duration=&       # Max minutes to seach forward.  Math works eg 2*24*60 = 2 days
#     zulu=true&       # default output in Zulu time as received, false = local time
#     expand=false&    # true = show full description copy. Default false, about 100 chars
#     limit=100        # limit to N results default 100, 0 means all
#     pid=false        # Display the SD ProgramID in description
# 
#   This is an AND search.  All conditions specified must be true to be a match
#   Copy can be used to find episode names or number, actor names, etc.
#   for Title & Copy search, its a sub-string match, not a word-match
#     so "nypd" finds nypd blue, but "Nypd  blue" does not (2 spaces)
#
# All keys do not need to be listed, only the ones you need
#
# PiGS internal schedule Search Examples
# Display the first 60 minutes of the PiGS internal schedule
#    /search?expand=&duration=60&startdate=&zulu=true&channel=&title=&copy=
#    
# Display full schedule for channel 3.1 
#    /search?expand=&duration=&startdate=&zulu=true&channel=3.1&title=&copy=
# Find all programs with the word 'fire' in the description
#    /search?expand=&duration=&startdate=&zulu=true&channel=&title=&copy=fire
# Actor search: find all programs with the word 'franz' in the description
#    /search?expand=&duration=&startdate=&zulu=true&channel=&title=&copy=franz
# Show name search: Find nypd blue
#    /search?expand=&duration=&startdate=&zulu=true&channel=&title=nypd&copy=
# Limit name search to DST change over 10/31 to 11/1 for channel 15.1 
#    /search?expand=&duration=2*24*60&startdate=2020-10-31T00:00:00Z&zulu=true&channel=15.1&title=&copy=
#     start 10/31, for 2 days (in minutes) = 2*24*60
#
# V1.13x 2020-Oct-19
#    These were Beta versions of 1.20
#
# V1.12 2020-Oct-05
#    Change LocalTimeStr() to work cross platform better.
#       Some display CDT, others Central Daylight Time was messing up DNS dipslay on /pigstatus
#    Fix dict init bug in ls.py lines 194-196
# V1.11 2020-Oct-03
#    Defaulted PinkPigPos to 0 for new installs.
#    Increased log file size to 150K before rollover.  Not quite enough look-back to solve issues
#    Add try/except in Builder calls to handle build exceptions, SD errors 
#    Add a 30 minute timer to retry the Service build if the Service build fails at boot
#    Fixes in HTML HEAD and unsupported commands
#    Added a DNS server to PiGS.  External DNS servers no longer needed.
#      sudo sysemctl disable dnsmasq, sudo sysemctl stop dnsmasq
#     /pigstatus Displays "DNS Enabled": "True, Last Request at 15:58:54(CDT)",
#    New Config Keys:
#      "IconType"     : 1,     # 0=No Icons, 1=Sched Direct, 2=CM/Rovi Icons, 3=Future???
#      "UseCMImages"  : key removed.  IconType takes its place
#      "PigsDoesDNS"  : True,  # PiGS acts as a DNS Server for DVR+.  Forward requests for msft,google,amzn, etc
#                                Defaults to False for upgrades so as not break an installation
#      "PigsDelayDNS" : True,  # PiGS starts DNS server thread after Schedule is built (PiGS ready for requests)
#      "DnsHosts"     : dict of "url" : "IP" host pairs for PiGS DNS server
#      "Server_IP"    : '0.0.0.0'
#          PiGS defaults to  0.0.0.0 listening on all active Interfaces eg eth0 wlan0 localhost 127.0.0.1
#          Use Server_IP to tell PiGS to listen on specific IP for port 80(web UI/epg requests) & 53(dns).
#          Sometimes needed in multiple NIC (wired/wifi) situations 
#
#    UI Updates
#    Added some link urls to SD & pigslog.txt, Added DNS enable checkbox and code behind it.
#    Added dropdown list for IconType / removed []Use CM Icons checkbox
#    Displayed listen IP near Version, updated the Help at bottom of page
#    Added note to remind user to do a dvr MGR on lineup or Icon changes
#    Added link to ./pigstatus on Config page
#
# V1.02rc 2020-Sep-28
#    A beta version of 1.10 sent to testers.  Decided to bump to 1.10 since DNS is a big change
#
# V1.01 2020-Sep-22
#    Change DaysLimit to 0 to fetch the maximum amount of days from SD (was 16)
#    Add request counter to GetLinearSchedule call, used Cnt ==90 to print Done msg
#    Add PiGS Server IP and Port to pigs.json to allow IP to be customized w/o editing the Python code
#     "Server_IP"    : "0.0.0.0",
#     "Server_Port"  : 8000,
#    Convert canadian zips to uppercase in sd.py and fix bug in ls.py where I accessed empty stationList []
#    At start PiGS verifies PiGS lineups match SD account (eg if someone used epg123 or other prog to mess up SD)
#    Added error checking for SD server down status code 3000 that occurred 9-21-2020
#
# V1.00 2020-Aug-14
#    Builder can now receive params, signal the caller, and return results
#    Moved AddRemoveLineups, Preview, and SDLogin into Builder
#    Builder is really the SD-Resource-Manager, but Builder is its name
#    Disabled UI Save and 'Verify Login' buttons while PiGS processes the Commands
#    Next Rebuild Time (23hrs) is in pigs.json "RebuildSecs" : 23*60*60,
#    set permissions to 777 for myicons,cmicons,sdicons when they are created 
#
# status command updated with Builder info: 192.168.1.200/pigstatus   returns
#{
#  "PiGS Start Time": "2020-08-09 17:10:53(CDT)",
#  "   PiGS Up-Time": "1 day, 13:41:45.134840",
#  "   PiGS_Version": "V0.9996 2020-Aug-07",
#  "  Builder Alive": true,
#  "Service Build Info": {
#    "Services_Built": "2020-08-10 07:07:45(CDT)",
#    "Total Channels": 32
#  },
#  "Schedule Build Info": {
#    "BuildStartTime": "2020-08-10 07:07:46(CDT)",
#    "Build_End_Time": "2020-08-10 07:08:56(CDT)",
#    "Build_Progress": "100%",
#    "DaysLimit": 16,
#    "Programs": 15948,
#    "SchedMBs": "15.7 MB",
#    "SchdEnds": "2020-08-25T23:30:00Z",
#    "NextBuildTime": "2020-08-11 16:26:31(CDT)",
#    "AutoBuildTime": "Not Scheduled."
#    },
#  "DVR Status List": "------------------------",
#  "192.168.1.202": {
#    "ListIndex": 15783,
#    "FirstRqst": "2020-08-10 17:26:31(CDT)",
#    "Last_Rqst": "2020-08-10 17:31:08(CDT)",
#    "__Elapsed": "04:37s"
#  },
#  "192.168.1.201": {
#    "ListIndex": 15783,
#    "FirstRqst": "2020-08-10 17:46:27(CDT)",
#    "Last_Rqst": "2020-08-10 17:50:51(CDT)",
#    "__Elapsed": "04:24s"
#  }
#}
#
# V0.9995 2020-Aug-07
#    PinkPigPos in pigs.json can be a row number (integer), or a channel number like 3.1 or "3.1", or 0 to disable
#    Added DvrBuildTimer Thread. Daily SD Build is scheduled 23 hours after the dvr does an update, or 1 hr before tomorrows update
#    Added WDogBuildTimer thread - its a watchdog timer in case a dvr didnt do its reqular update
#    Added more to the pigstatus Command.  <your-pigs-IP>/pigstatus
#    You can trigger a SD schedule build by going to the UI and clicking 'Save Settings'  Dont have to change anything
#    UI change: Lineup Preview can take up to 60 seconds at SD.  Disabled Preview button during this time.
#
# V0.9991 2020-Aug-01
#    Implemented Threading for Schedule Build to eliminate the "Broken pipe errors."
#    PiGS now responds to every web command with no delay (ex Preview, which can have a 60s delay).
#    If PiGS is busy building the scehdule and cant execute the request, PiGS returns a 503 server busy http error
#    The <PiGS IP>/pigstatus browser command now displays the build progress %, like 63%
#    Cleanup & imporove the Status Command
#    Mods to sv.py and cm.py to support Canadian style zip codes and CM icons
#    Change in sv.py test to support channel schedules on 100+ channels (Channel remapping in pigs.json)
#
# V0.999 2020-Jul-22
#    Support of Canadian 6-digit zipcodes (means UI has to accept alpha-numeric zips)
#    Removed ("Connection", "Keep-Alive") http header from responses to DVR+
#      This was in the Wire-shark responses to DVR+ from the CM Servers, but I don't think it is necessary
#      and some users were experiencing an occasional "Broken Pipe" error.  I could see them under certain conditions
#      The errors seem to go away for me when I removed that header.
#    Blank users SD password on UI after successful login for security
#    Renamed log file to pigslog.txt so the logfile will display in browser <your-pigs-ip>/pigslog.txt
#     Beta testers should/could edit there pigs.json files and change "LoggingFile" to pigslog.txt
#    In BuildLinearSchedule() search for 'USA Parental Rating' even though that appears to be the only one provided
#    Shorten Cast List: to Cast: to save a few description chars
#
#    Added browser method <your-pigs-ip>/pigstatus to monitor PiGS status and & DVR states.
#    You can use this to verify PiGS is running on your computer or Pi and as a 
#     'Replacement' for the Last Internet Guide update on the DVR+ StbHealth Menu.
#     This can be refreshed in your browser while updates are on-going and works with multiple DVR's
#    192.168.1.200/pigstatus    for me  returns
#    {
#      "PiGS Start Time": "2020-07-18 09:57:05(CDT)",
#          PiGS Up-Time": "2 days, 3:00:54.147750",
#      "   PiGS_Version": "V0.999 2020-Jul-18",
#      " Services Built": "2020-07-18 09:57:05(CDT)",
#      " Schedule Built": "2020-07-18 14:50:30(CDT)",
#      "Schedule Build Info": {
#        "BuildStartTime": "2020-07-22 20:02:26(CDT)",
#        "DaysLimit": 16,
#         "Programs": 16314,
#         "SchedMBs": "16.2 MB",
#         "SchdEnds": "2020-08-07T23:30:00Z"
#      },
#      "DVR Status List": "------------------------",
#      "192.168.1.201": {
#        "ListIndex": 14798,
#        "FirstRqst": "2020-07-18 16:11:44(CDT)",
#        "Last_Rqst": "2020-07-18 16:17:37(CDT)"
#        "__Elapsed": "05:59s"
#      },
#      "192.168.1.203": {
#        "ListIndex": 14938,
#        "FirstRqst": "2020-07-18 19:40:30(CDT)",
#        "Last_Rqst": "2020-07-18 19:46:12(CDT)"
#      }
#    }
#
# V0.998 2020-Jul-14 
#    Added ut.py for misc utility functions used accross modules
#    Added ReleaseNotes.txt to archive all these changes
#    Handled a lineup preview status message better.  Requests are sometimes Queued on SD server for generation
#    Beta testers should increase DaysLimit to 16 from 14 to prevent DVR+ 403 errors
#    Got rid of extra logging 'noise' in LinearScheduleBuild() and reported the RF channel PiGS finds for each channel
#    Logged Platform Info (OS, 32/64-bit, versions, etc) at start for debugging. Pi's are 32-bit native machines.
#    Modified pigs.service for a default install directory of ./pigs
#    Removed the (cnt) from the end of the program description
#
#    Added support to resize the very large SD channel image icons
#    The actual icon display area on the DVR+ appears to be 92w x 51h pixels
#    Added Pillow imaging lib but wrapped it in a try/except so it should not crash anyones system
#    Added new keys to pigs.json to control SD Image resizing and for experimentation
#     "CacheCMIcons" : True, to the config to control caching of CM icons. 
#     "CacheSDIcons" : False, to the config to control caching/resizing of SD icons.
#     "SaveFsSDIcons": False, # Saves a copy of the FullSize SD icons in ./sdicons for external manipulation
#     "SDImSize"     : [72,54], # 360w x 270 / 5 = 72w x 54h (json cant do Tuple, so will use a list)
#     "SDImResample" : 1,     # Deault to LANCZOS.   Pillow Lib resize() Resample type 0..5
#     "SDImReduceGap": None,  # Pillow 7.0 Reducing Gap 
#     PiGS attempts to resize the SD icons and cache them, but they looked pretty coarse might need work
#
#    Moderate change...  Changed Service List build to lookup StationID's by callsign instead of channel number.
#      to ensure PiGS gets the channel the user selected.  some markets have two 14.1's for example.
#    Also, Since PiGS is multi-lineup, we need to verify that PiGS is building lists without duplicate chan #'s
#    Can DVR+ pull in two RF channels with the same Channel number?  Yes.  Is the lower RF and first channel? No?
#     Does it consistently add 100 to the channel number?  Seen during repack, rescan.
#    DVR+ seems to place Dup channels if finds during its channel scan starting at 100. a dup 3.1 would be at 100.1
#    PiGS should be able to provide guide data for these channels if we manually make an entry in pigs.json
#      as long as we can say - its the same as XXXX then it should all just work.
#    In service build, check for and skip duplicate channels, with a log warning.
#    UI also tests for duplicate channels.  Needed in case a lineup has two channels with same # (eg 78628)
#      or if user picks two nearby zipcodes and selects the same channel from both lineups.
#    Maybe what I want is - no duplicate callsigns...
#    I guess I dont know if DVR+ matches the guide entries by callsign or channel #.  Must be Ch #.
#    This part needs more work with some real-life situations and testing
#
# V0.997 2020-Jul-xx sent to ej_eddie for image experiments
#
# V0.996 2020-Jul-04
#    Found big bug in DVR call to GetServiceDetails() from some reports of flaky channel icon behavior.
#    The channel lineup list was being extended (duplicated) each time DVR requested the list (eg Manual Guide Refresh)
#    If the user was going back and forth between Rovi & CM icons, Saving and doing MGR's, DVR would receive multiple
#    channel 3.1's with different icon urls.  The may have been very confusing to DVR+.
#    If you did not reboot PiGS for a while and did many MGR's there would be many copies of each channel in the lineup
#    but there can only be 1 definition per channel.
#    This bug has been in there since the very beginning and had nothing to do with adding the SD icons.
#    But with the addition of the SD icons, the problem became more visible to users with missing or duplicated icons.
#    It was a 1-line of code fix.  Clear the old list before adding in the new list
#
# V0.995   2020-Jul-04
#    Added total_size() to compute size of MasterAiringList[] (biggest object in PiGS)
#      With 2 full lineups 53593 and 19503, 191 channels selected: 
#          92,737 total programs, the MasterAiringList size 86.5 MB
#      With just my 29 channels 53593, MasterAiringList size 13.2 MB
#      A very rough estimate: it appears about 0.5MB of RAM is needed per channel selected
#       e.g. for 100 channels, 50MB of RAM would be used (peaks higher during the build)
#      PiGS just might run on a Pi-Zero Wireless!
#
# V0.994 2020-Jul-03
#    BUG FIX - in the full SD icon support implementation - possible random image links
#       added a copy.copy() to ServiceListBuild  img0["ImageUrl"] = copy.copy(IconUrl)
#    Made the UI "Save Settings" button bigger and Blue to make it stand out
#    Disabled SD Icon Theme Control when CM/Rovi Icon is checked
#    Spelling fixes and more help text on the UI config page
#
# V0.993 2020-Jul-02
#    Schedules Direct Channel Icons are working!  Two new keys have been added to cfg
#     "SdIconTheme"  : "dark", # choices are dark, gray, light, white
#     "DefaultIcon"  : "pinkpig.png",  # used if no guide icon found. another could be "92x36box.png"   
#    Added a drop down to the UI for SD Icon Theme
#    Beta user can rename their icons dir to cmicons
#    Icon strategy is a little different in this verion....
#       ./myicons holds user custom icons, use this icon if one is found
#       ./cmicons caches CM/Rovi icons 
#       SD icons are used directly from their url (live fetch)
#       If no icon can be found from any of the above, use the file from "DefaultIcon"
#
#    SD does not provide TimeZone info with their data, but TZ info it is in the CM data
#    I realized TZ is local to PiGS, your DVR, so I added  "TimeZones" : []  to pigs.json and attempted
#    to initialize it to the users local time - we'll see what happens in Nov
#    Try this link to see TimeZone info
#    http://proxy-elb-1800886808.us-west-2.elb.amazonaws.com/TVlistings/v9/listings/services/postalcode/53593/info?apikey=v4edzwwtca4e847n72j8kesw&locale=en-US&countrycode=US&msoid=306430&format=json
#    PiGS will use whatever is in this key to update the meassages to DVR+
#    This key might allow us to experiment at Time Change without code modification if needed???
#
# V0.992 2020-Jun-28
#    CM servers went down 2020-06-27 causing PiGS to crash on Services build.
#    Improved error handling in CM.py  CM sends http=200, with an error block NPE
#    Had to interpret that block to see if there was an error code NPE, then handle errors.
#
#    I didnt like the crash, and I didnt like not having my icons, so....
#    PiGS now caches the CM Icons in ./icons
#    if the dir ./icons does not exist, PiGS creates it
#
#    Channel Icon Strategy.
#    1) look in ./icons/ for a icon file named <ChNum>.png, like 3.1.png
#       If found, use it.  if not, then...
#    2) If 'UseCMImages' is true, then get the CM image and save it as <ChNum>.png
#       Now same as #1. only way to update cm icons would be to delete the png's
#    3) If nothing found, then use my empty box icon. or pink pig icon
#    4) User can edit these icon files and put whatever they want.
#    This strategy allows a user to create their own channel Icons
#    https://tvschedule.zap2it.com/?aid=gapzap has nice 55wx41h images
#    I downloaded one of those 'bigger icons for nbc', renamed it 15.1.png
#    copied it to ./icons and dvr+ displayed the bigger icon!
#    I did a double sized image test and found DVR+ resizes images it receives.
#
#    Could also get the SD images, but their icons are boring - you are better off borrowing from zap2it
#
# V0.991 2020-Jun-27  Only sent to Frank70 for Multi-DVR schedule testing
#    In pc.py (the CM data scanner), Printed Channel Master TimeZone object in Prep for November
#      I dont know what the TZ info does for DVR+, but SD has no TZ info in their data.
#      We will have to watch what happens in Nov. Both SD & CM data is in Zulu, so hopefully nothing.
#    Fix bug in ls.py GetLinearSchedule() - multiple DVRs making requests at same time.
#      GetLinearSchedule() now saves a list of dvr's (keyed on their IP) making requests and
#      maintains the 'Request State' (ListIndex and Starting time) between calls.
#      Search backwards is still there to ensure a valid search starting point.
#    Added a 'RotatingFileHandler' to PiGS logging.  Previously I just deleted it at >250K in size.
#      that was a poor solution b/c if you never reboot, it never got deleted and would get HUGE
#      now its handled by the logger module.  when a log file reaches 100K, it starts a new logfile 
#      https://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/
#      -rw-r--r-- 1 root root   128 Jun 26 08:11 pigs.log    <<< current log file (my test 1k size)
#      -rw-r--r-- 1 root root   960 Jun 26 08:10 pigs.log.1
#      -rw-r--r-- 1 root root   960 Jun 26 08:09 pigs.log.3  <<< oldest log file, up to 3.
#
# V0.99 2020-Jun-21
#    PiGS is no longer working directory dependent. thanks ej_eddie for suggestion
#      PiGS sets its working directory to the location of pigs.py 
#    Print version string to logfile at warning level if case user is above INFO level
#    Should I bind to 0.0.0.0 or another IP address?
#    Reworked pc.py, the CM data integrity scanner and desktop tester, for cmd line usage
#        sudo python3 pc.py <port> <zipcode-to-scan>
#     eg sudo python3 pc.py 80 53593   or if pigs is running, sudo python3 pc.py 8080 53593
#    Added GNU GPL V3 license
#
# V0.98 2020-Jun-21
#       Added UserAgent header to SD calls
#       Skipped SD token and ServiceList build if username/pwd not yet configured
#       Changed logging level on some messages to Info
#       Remvoved json.JSONDecodeError: exception at config load to support Python 3.4
#       Added Support for PiGS starting at boot: up to 30s wait loop so Network can come online
#          PiGS will probably be running before the Pi desktop shows up
#          I wonder if my method of testing the IP works cross platform????
#       If network never becomes avaiable, then skip the build of ServicesList at Start
#         and add checks to see if it is built in Zipcode and Schedule gets
#       Included 'pigs.service' with instructions in file to get miss piggy to start at boot
#       Added Pigs.desktop, a file to make a desktop icon to start PiGS manually
#
# V0.97 2020-Jun-17
#       Reset 'ListIndex' to 0 after schedule build
#       This version sent out to a few people for beta testing
#       more cleanup in pc.py, Printed PiGS IP when app starts
#       remove printing users clear password from log file
#       Created a LocalTimeStr() function that uses system info to display LT
#       Fix bug in description len 512 counting
#       if logfile does not exist, create it and set permissions to 0o666
#       change logging so it does not go to file when run from Dev Environment
#
# V0.96 2020-Jun-11
#       made pinkpig.png transparent
#       cleanup in cm.py and pc.py, added some pc.py scan features & log to cmdata.log
#       Save config file pigs.json on exit to preserve build dates and new version
#       Cleanup on index.htm help text at bottom, more SD login error checking.
#
# V0.95 2020-Jun-07
#       Logging to pigs.log and Info logs to console
#       Added CC,HD,Ratings, type flags to listings
#       Add (New) to start of description since dvr+ has a bug showing New status
#           when using Info when the Guide is displayed
#       Removed duplicate programIDs before requesting program details from SD
#