builder: release-mozilla-beta-firefox_publish_balrog slave: bld-linux64-spot-304 starttime: 1490970315.87 results: failure (2) revision: 0a36cc120d6e7b8bc19176f5efac6b00dd9ad072 ========= Started set props: master (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:15.867346) ========= master: http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/ ========= Finished set props: master (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:15.867865) ========= ========= Started downloading to buildprops.json (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:15.868159) ========= ========= Finished downloading to buildprops.json (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:15.947833) ========= ========= Started 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:15.948169) ========= rm -rf scripts properties in dir /builds/slave/rel-m-beta-fx_publish_balrog-0/. (timeout 1200 secs) watching logfiles {} argv: ['rm', '-rf', 'scripts', 'properties'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-304.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_publish_balrog-0 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.011275 ========= master_lag: 0.04 ========= ========= Finished 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:15.999114) ========= ========= Started 'bash -c ...' (results: 0, elapsed: 1 secs) (at 2017-03-31 07:25:15.999471) ========= bash -c 'python /tools/checkouts/build-tools/buildfarm/utils/archiver_client.py mozharness --repo releases/mozilla-beta --rev 0a36cc120d6e7b8bc19176f5efac6b00dd9ad072 --destination scripts --debug' in dir /builds/slave/rel-m-beta-fx_publish_balrog-0/. (timeout 1200 secs) watching logfiles {} argv: ['bash', '-c', u'python /tools/checkouts/build-tools/buildfarm/utils/archiver_client.py mozharness --repo releases/mozilla-beta --rev 0a36cc120d6e7b8bc19176f5efac6b00dd9ad072 --destination scripts --debug'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-304.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_publish_balrog-0 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False 2017-03-31 07:25:16,059 truncating revision to first 12 chars 2017-03-31 07:25:16,059 Setting DEBUG logging. 2017-03-31 07:25:16,059 attempt 1/10 2017-03-31 07:25:16,060 Getting archive location from https://api.pub.build.mozilla.org/archiver/hgmo/releases/mozilla-beta/0a36cc120d6e?&preferred_region=us-west-2&suffix=tar.gz&subdir=testing/mozharness 2017-03-31 07:25:17,365 unpacking tar archive at: mozilla-beta-0a36cc120d6e/testing/mozharness/ program finished with exit code 0 elapsedTime=1.463866 ========= master_lag: 0.01 ========= ========= Finished 'bash -c ...' (results: 0, elapsed: 1 secs) (at 2017-03-31 07:25:17.474364) ========= ========= Started set props: script_repo_revision (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:17.474711) ========= script_repo_revision: 0a36cc120d6e7b8bc19176f5efac6b00dd9ad072 ========= Finished set props: script_repo_revision (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:17.475165) ========= ========= Started downloading to oauth.txt (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:17.475446) ========= ========= Finished downloading to oauth.txt (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:17.483981) ========= ========= Started '/tools/buildbot/bin/python scripts/scripts/release/publish_balrog.py ...' failed (results: 2, elapsed: 2 secs) (at 2017-03-31 07:25:17.484250) ========= /tools/buildbot/bin/python scripts/scripts/release/publish_balrog.py -c releases/updates_firefox_beta.py in dir /builds/slave/rel-m-beta-fx_publish_balrog-0/. (timeout 3600 secs) (maxTime 14400 secs) watching logfiles {} argv: ['/tools/buildbot/bin/python', 'scripts/scripts/release/publish_balrog.py', '-c', 'releases/updates_firefox_beta.py'] environment: CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_HASHDIR= CCACHE_UMASK=002 DISPLAY=:2 G_BROKEN_FILENAMES=1 HG_SHARE_BASE_DIR=/builds/hg-shared HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-304.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LC_ALL=C LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld MOZ_AUTOMATION=1 MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin PROPERTIES_FILE=/builds/slave/rel-m-beta-fx_publish_balrog-0/buildprops.json PWD=/builds/slave/rel-m-beta-fx_publish_balrog-0 SHELL=/bin/bash SHLVL=1 TERM=linux TINDERBOX_OUTPUT=1 TMOUT=86400 TOOLTOOL_CACHE=/builds/tooltool_cache TOOLTOOL_HOME=/builds USER=cltbld _=/tools/buildbot/bin/python using PTY: False 07:25:17 INFO - MultiFileLogger online at 20170331 07:25:17 in /builds/slave/rel-m-beta-fx_publish_balrog-0 07:25:17 INFO - Using buildbot properties: 07:25:17 INFO - { 07:25:17 INFO - "channels": "beta", 07:25:17 INFO - "product": "firefox", 07:25:17 INFO - "build_number": 1, 07:25:17 INFO - "taskId": "9vnS5yRjQqSXS-qew3LDVw", 07:25:17 INFO - "repository": "", 07:25:17 INFO - "buildername": "release-mozilla-beta-firefox_publish_balrog", 07:25:17 INFO - "buildnumber": 11, 07:25:17 INFO - "balrog_api_root": "https://aus4-admin.mozilla.org/api", 07:25:17 INFO - "basedir": "/builds/slave/rel-m-beta-fx_publish_balrog-0", 07:25:17 INFO - "project": "", 07:25:17 INFO - "platform": null, 07:25:17 INFO - "version": "53.0b8", 07:25:17 INFO - "release_promotion": true, 07:25:17 INFO - "script_repo_revision": "0a36cc120d6e7b8bc19176f5efac6b00dd9ad072", 07:25:17 INFO - "branch": "mozilla-beta", 07:25:17 INFO - "master": "http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/", 07:25:17 INFO - "revision": "0a36cc120d6e7b8bc19176f5efac6b00dd9ad072", 07:25:17 INFO - "slavename": "bld-linux64-spot-304", 07:25:17 INFO - "repo_path": "releases/mozilla-beta" 07:25:17 INFO - } 07:25:17 INFO - Overriding product with firefox 07:25:17 INFO - Overriding version with 53.0b8 07:25:17 INFO - Overriding build_number with 1 07:25:17 INFO - Overriding channels with beta 07:25:17 INFO - Overriding balrog_api_root with https://aus4-admin.mozilla.org/api 07:25:17 INFO - Run as scripts/scripts/release/publish_balrog.py -c releases/updates_firefox_beta.py 07:25:17 INFO - Dumping config to /builds/slave/rel-m-beta-fx_publish_balrog-0/logs/localconfig.json. 07:25:17 INFO - {'append_to_log': False, 07:25:17 INFO - 'archive_domain': 'archive.mozilla.org', 07:25:17 INFO - 'archive_prefix': 'https://archive.mozilla.org/pub', 07:25:17 INFO - 'balrog_api_root': u'https://aus4-admin.mozilla.org/api', 07:25:17 INFO - 'balrog_url': 'https://aus5.mozilla.org', 07:25:17 INFO - 'balrog_use_dummy_suffix': False, 07:25:17 INFO - 'balrog_username': 'ffxbld', 07:25:17 INFO - 'base_work_dir': '/builds/slave/rel-m-beta-fx_publish_balrog-0', 07:25:17 INFO - 'build_number': 1, 07:25:17 INFO - 'buildbot_json_path': 'buildprops.json', 07:25:17 INFO - 'channels': u'beta', 07:25:17 INFO - 'config_files': ('releases/updates_firefox_beta.py',), 07:25:17 INFO - 'credentials_file': 'oauth.txt', 07:25:17 INFO - 'download_domain': 'download.mozilla.org', 07:25:17 INFO - 'ignore_no_changes': True, 07:25:17 INFO - 'log_level': 'info', 07:25:17 INFO - 'log_name': 'updates_beta', 07:25:17 INFO - 'log_to_console': True, 07:25:17 INFO - 'opt_config_files': (), 07:25:17 INFO - 'previous_archive_prefix': 'https://archive.mozilla.org/pub', 07:25:17 INFO - 'product': u'firefox', 07:25:17 INFO - 'push_dest': 'ssh://hg.mozilla.org/build/tools', 07:25:17 INFO - 'repo': {'branch': 'default', 07:25:17 INFO - 'dest': 'tools', 07:25:17 INFO - 'repo': 'https://hg.mozilla.org/build/tools', 07:25:17 INFO - 'vcs': 'hg'}, 07:25:17 INFO - 'shipped-locales-url': 'https://hg.mozilla.org/releases/mozilla-beta/raw-file/{revision}/browser/locales/shipped-locales', 07:25:17 INFO - 'ssh_key': '~/.ssh/ffxbld_rsa', 07:25:17 INFO - 'ssh_user': 'ffxbld', 07:25:17 INFO - 'update_channels': {'beta': {'channel_names': ('beta', 07:25:17 INFO - 'beta-localtest', 07:25:17 INFO - 'beta-cdntest'), 07:25:17 INFO - 'mar_channel_ids': (), 07:25:17 INFO - 'patcher_config': 'mozBeta-branch-patcher2.cfg', 07:25:17 INFO - 'publish_rules': ('firefox-beta',), 07:25:17 INFO - 'requires_mirrors': True, 07:25:17 INFO - 'rules_to_update': ('firefox-beta-cdntest', 07:25:17 INFO - 'firefox-beta-localtest'), 07:25:17 INFO - 'update_verify_channel': 'beta-localtest', 07:25:17 INFO - 'version_regex': '^(\\d+\\.\\d+(b\\d+)?)$'}}, 07:25:17 INFO - 'vcs_share_base': '/builds/hg-shared', 07:25:17 INFO - 'version': u'53.0b8', 07:25:17 INFO - 'volatile_config': {'actions': None, 'add_actions': None, 'no_actions': None}, 07:25:17 INFO - 'work_dir': 'build'} 07:25:17 INFO - [mozharness: 2017-03-31 14:25:17.632654Z] Running clobber step. 07:25:17 INFO - Running main action method: clobber 07:25:17 INFO - rmtree: /builds/slave/rel-m-beta-fx_publish_balrog-0/build 07:25:17 INFO - retry: Calling rmtree with args: ('/builds/slave/rel-m-beta-fx_publish_balrog-0/build',), kwargs: {}, attempt #1 07:25:17 INFO - [mozharness: 2017-03-31 14:25:17.673786Z] Finished clobber step (success) 07:25:17 INFO - [mozharness: 2017-03-31 14:25:17.674053Z] Running pull step. 07:25:17 INFO - Running main action method: pull 07:25:17 INFO - mkdir: /builds/slave/rel-m-beta-fx_publish_balrog-0/build 07:25:17 INFO - Changing directory to /builds/slave/rel-m-beta-fx_publish_balrog-0/build. 07:25:17 INFO - retry: Calling _get_revision with args: (, 'tools'), kwargs: {}, attempt #1 07:25:17 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', '--version'] 07:25:17 INFO - Copy/paste: hg --config ui.merge=internal:merge --version 07:25:17 INFO - Mercurial Distributed SCM (version 3.9.1) 07:25:17 INFO - (see https://mercurial-scm.org for more information) 07:25:17 INFO - Copyright (C) 2005-2016 Matt Mackall and others 07:25:17 INFO - This is free software; see the source for copying conditions. There is NO 07:25:17 INFO - warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 07:25:17 INFO - Return code: 0 07:25:17 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'debuginstall'] 07:25:17 INFO - Copy/paste: hg --config ui.merge=internal:merge debuginstall 07:25:18 INFO - checking encoding (ascii)... 07:25:18 INFO - checking Python executable (/tools/python27/bin/python2.7) 07:25:18 INFO - checking Python version (2.7.3) 07:25:18 INFO - checking Python lib (/tools/python27/lib/python2.7)... 07:25:18 INFO - checking Mercurial version (3.9.1) 07:25:18 INFO - checking Mercurial custom build () 07:25:18 INFO - checking module policy (c) 07:25:18 INFO - checking installed modules (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial)... 07:25:18 INFO - checking templates (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/templates)... 07:25:18 INFO - checking default template (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/templates/map-cmdline.default) 07:25:18 INFO - checking commit editor... (vi) 07:25:18 INFO - checking username (Mozilla Release Engineering ) 07:25:18 INFO - no problems detected 07:25:18 INFO - Return code: 0 07:25:18 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', '--config', 'extensions.robustcheckout=/builds/slave/rel-m-beta-fx_publish_balrog-0/scripts/external_tools/robustcheckout.py', 'robustcheckout', 'https://hg.mozilla.org/build/tools', 'tools', '--sharebase', '/builds/hg-shared', '--branch', 'default'] 07:25:18 INFO - Copy/paste: hg --config ui.merge=internal:merge --config extensions.robustcheckout=/builds/slave/rel-m-beta-fx_publish_balrog-0/scripts/external_tools/robustcheckout.py robustcheckout https://hg.mozilla.org/build/tools tools --sharebase /builds/hg-shared --branch default 07:25:18 INFO - ensuring https://hg.mozilla.org/build/tools@default is available at tools 07:25:18 INFO - warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info 07:25:19 INFO - (sharing from existing pooled repository 7ae7fb134bf7aec6ec96a062ff47a69053dd2973) 07:25:19 INFO - searching for changes 07:25:19 INFO - no changes found 07:25:19 INFO - (pulling to obtain default) 07:25:19 INFO - warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info 07:25:19 INFO - (remote resolved default to 0a0b27078a1856d43a068f4fabd12a536c533f78; result is not deterministic) 07:25:19 INFO - (revision already present locally; not pulling) 07:25:19 INFO - 971 files updated, 0 files merged, 0 files removed, 0 files unresolved 07:25:19 INFO - updated to 0a0b27078a1856d43a068f4fabd12a536c533f78 07:25:19 INFO - Return code: 0 07:25:19 INFO - Changing directory to /builds/slave/rel-m-beta-fx_publish_balrog-0. 07:25:19 INFO - [mozharness: 2017-03-31 14:25:19.717382Z] Finished pull step (success) 07:25:19 INFO - [mozharness: 2017-03-31 14:25:19.717575Z] Running submit-to-balrog step. 07:25:19 INFO - Running main action method: submit_to_balrog 07:25:19 INFO - retry: Calling with args: (), kwargs: {}, attempt #1 07:25:19 INFO - Running command: ['python', '/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/balrog-release-shipper.py', '--api-root', u'https://aus4-admin.mozilla.org/api', '--credentials-file', '/builds/slave/rel-m-beta-fx_publish_balrog-0/oauth.txt', '--username', 'ffxbld', '--version', u'53.0b8', '--product', u'firefox', '--build-number', '1', '--verbose', '--rules', 'firefox-beta'] 07:25:19 INFO - Copy/paste: python /builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/balrog-release-shipper.py --api-root https://aus4-admin.mozilla.org/api --credentials-file /builds/slave/rel-m-beta-fx_publish_balrog-0/oauth.txt --username ffxbld --version 53.0b8 --product firefox --build-number 1 --verbose --rules firefox-beta 07:25:19 INFO - Traceback (most recent call last): 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/balrog-release-shipper.py", line 14, in 07:25:19 INFO - from balrog.submitter.cli import ReleasePusher, ReleaseScheduler 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/balrog/submitter/cli.py", line 1, in 07:25:19 INFO - import arrow 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/__init__.py", line 3, in 07:25:19 INFO - from .arrow import Arrow 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/arrow.py", line 11, in 07:25:19 INFO - from dateutil import tz as dateutil_tz 07:25:19 INFO - ImportError: No module named dateutil 07:25:19 ERROR - Return code: 1 07:25:19 ERROR - 1 not in success codes: [0] 07:25:19 WARNING - setting return code to 2 07:25:19 FATAL - Halting on failure while running ['python', '/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/balrog-release-shipper.py', '--api-root', u'https://aus4-admin.mozilla.org/api', '--credentials-file', '/builds/slave/rel-m-beta-fx_publish_balrog-0/oauth.txt', '--username', 'ffxbld', '--version', u'53.0b8', '--product', u'firefox', '--build-number', '1', '--verbose', '--rules', 'firefox-beta'] 07:25:19 FATAL - Running post_fatal callback... 07:25:19 FATAL - Exiting 2 07:25:19 INFO - [mozharness: 2017-03-31 14:25:19.746637Z] Finished submit-to-balrog step (failed) 07:25:19 INFO - Running post-run listener: copy_logs_to_upload_dir 07:25:19 INFO - Copying logs to upload dir... 07:25:19 INFO - mkdir: /builds/slave/rel-m-beta-fx_publish_balrog-0/build/upload/logs program finished with exit code 2 elapsedTime=2.256732 ========= master_lag: 0.01 ========= ========= Finished '/tools/buildbot/bin/python scripts/scripts/release/publish_balrog.py ...' failed (results: 2, elapsed: 2 secs) (at 2017-03-31 07:25:19.752784) ========= ========= Started no change (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:19.754006) ========= bash -c 'for file in `ls -1`; do cat $file; done' in dir /builds/slave/rel-m-beta-fx_publish_balrog-0/properties (timeout 1200 secs) watching logfiles {} argv: ['bash', '-c', 'for file in `ls -1`; do cat $file; done'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-304.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_publish_balrog-0/properties SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.005022 ========= master_lag: 0.04 ========= ========= Finished no change (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:19.802114) ========= ========= Started 'rm -f ...' (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:19.802439) ========= rm -f oauth.txt in dir /builds/slave/rel-m-beta-fx_publish_balrog-0/. (timeout 1200 secs) watching logfiles {} argv: ['rm', '-f', 'oauth.txt'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-304.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_publish_balrog-0 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.002433 ========= master_lag: 0.04 ========= ========= Finished 'rm -f ...' (results: 0, elapsed: 0 secs) (at 2017-03-31 07:25:19.849779) ========= ========= Started reboot skipped (results: 3, elapsed: 0 secs) (at 2017-03-31 07:25:19.850094) ========= ========= Finished reboot skipped (results: 3, elapsed: 0 secs) (at 2017-03-31 07:25:19.850378) ========= ========= Total master_lag: 0.15 =========