builder: release-mozilla-beta-firefox_chcksms slave: bld-linux64-spot-376 starttime: 1491619258.47 results: failure (2) revision: 65bd52f60c9f ========= Started set props: master (results: 0, elapsed: 0 secs) (at 2017-04-07 19:40:58.468374) ========= master: http://buildbot-master91.bb.releng.usw2.mozilla.com:8001/ ========= Finished set props: master (results: 0, elapsed: 0 secs) (at 2017-04-07 19:40:58.468819) ========= ========= Started downloading to buildprops.json (results: 0, elapsed: 0 secs) (at 2017-04-07 19:40:58.469107) ========= ========= Finished downloading to buildprops.json (results: 0, elapsed: 0 secs) (at 2017-04-07 19:40:58.538257) ========= ========= Started 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2017-04-07 19:40:58.538557) ========= rm -rf scripts properties in dir /builds/slave/rel-m-beta-fx_chcksms-00000000/. (timeout 1200 secs) watching logfiles {} argv: ['rm', '-rf', 'scripts', 'properties'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-376.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_chcksms-00000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.003593 ========= master_lag: 0.05 ========= ========= Finished 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2017-04-07 19:40:58.587632) ========= ========= Started 'bash -c ...' (results: 0, elapsed: 1 secs) (at 2017-04-07 19:40:58.587964) ========= bash -c 'python /tools/checkouts/build-tools/buildfarm/utils/archiver_client.py mozharness --repo releases/mozilla-beta --rev 65bd52f60c9f --destination scripts --debug' in dir /builds/slave/rel-m-beta-fx_chcksms-00000000/. (timeout 1200 secs) watching logfiles {} argv: ['bash', '-c', u'python /tools/checkouts/build-tools/buildfarm/utils/archiver_client.py mozharness --repo releases/mozilla-beta --rev 65bd52f60c9f --destination scripts --debug'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-376.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_chcksms-00000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False 2017-04-07 19:40:58,629 Setting DEBUG logging. 2017-04-07 19:40:58,629 attempt 1/10 2017-04-07 19:40:58,629 Getting archive location from https://api.pub.build.mozilla.org/archiver/hgmo/releases/mozilla-beta/65bd52f60c9f?&preferred_region=us-west-2&suffix=tar.gz&subdir=testing/mozharness 2017-04-07 19:40:59,844 unpacking tar archive at: mozilla-beta-65bd52f60c9f/testing/mozharness/ program finished with exit code 0 elapsedTime=1.583084 ========= master_lag: 0.01 ========= ========= Finished 'bash -c ...' (results: 0, elapsed: 1 secs) (at 2017-04-07 19:41:00.180385) ========= ========= Started set props: script_repo_revision (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.180711) ========= script_repo_revision: 65bd52f60c9f ========= Finished set props: script_repo_revision (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.181085) ========= ========= Started remove old nonce (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.181379) ========= rm -f nonce in dir /builds/slave/rel-m-beta-fx_chcksms-00000000/. (timeout 1200 secs) watching logfiles {} argv: ['rm', '-f', 'nonce'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-376.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_chcksms-00000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.003108 ========= master_lag: 0.04 ========= ========= Finished remove old nonce (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.228866) ========= ========= Started downloading to token (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.229192) ========= Slave: bld-linux64-spot-376 IP: 10.132.55.55 Duration: 25200 URI: https://mac-v2-signing3.srv.releng.scl3.mozilla.com:9120/token ========= Finished downloading to token (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.453032) ========= ========= Started set props: toolsdir (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.453452) ========= bash -c pwd in dir /tools/checkouts/build-tools (timeout 1200 secs) watching logfiles {} argv: ['bash', '-c', 'pwd'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-376.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/tools/checkouts/build-tools SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False /tools/checkouts/build-tools program finished with exit code 0 elapsedTime=0.003266 toolsdir: '/tools/checkouts/build-tools' ========= master_lag: 0.04 ========= ========= Finished set props: toolsdir (results: 0, elapsed: 0 secs) (at 2017-04-07 19:41:00.500382) ========= ========= Started '/tools/buildbot/bin/python scripts/scripts/release/generate-checksums.py ...' failed (results: 2, elapsed: 3 mins, 39 secs) (at 2017-04-07 19:41:00.500687) ========= /tools/buildbot/bin/python scripts/scripts/release/generate-checksums.py --stage-product firefox --bucket-name-full net-mozaws-prod-delivery-firefox --credentials /builds/release-s3.credentials --tools-repo /tools/checkouts/build-tools in dir /builds/slave/rel-m-beta-fx_chcksms-00000000/. (timeout 3600 secs) (maxTime 14400 secs) watching logfiles {} argv: ['/tools/buildbot/bin/python', 'scripts/scripts/release/generate-checksums.py', '--stage-product', 'firefox', '--bucket-name-full', 'net-mozaws-prod-delivery-firefox', '--credentials', '/builds/release-s3.credentials', '--tools-repo', '/tools/checkouts/build-tools'] environment: CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_HASHDIR= CCACHE_UMASK=002 DISPLAY=:2 G_BROKEN_FILENAMES=1 HG_SHARE_BASE_DIR=/builds/hg-shared HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-376.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LC_ALL=C LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld MOZ_AUTOMATION=1 MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox MOZ_SIGNING_SERVERS=gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120,gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120,gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120,macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120 MOZ_SIGN_CMD=python /tools/checkouts/build-tools/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-fx_chcksms-00000000/signing_cache -t /builds/slave/rel-m-beta-fx_chcksms-00000000/token -n /builds/slave/rel-m-beta-fx_chcksms-00000000/nonce -c /tools/checkouts/build-tools/release/signing/host.cert -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120 PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin PROPERTIES_FILE=/builds/slave/rel-m-beta-fx_chcksms-00000000/buildprops.json PWD=/builds/slave/rel-m-beta-fx_chcksms-00000000 SHELL=/bin/bash SHLVL=1 TERM=linux TINDERBOX_OUTPUT=1 TMOUT=86400 TOOLTOOL_CACHE=/builds/tooltool_cache TOOLTOOL_HOME=/builds USER=cltbld _=/tools/buildbot/bin/python using PTY: False 19:41:00 INFO - MultiFileLogger online at 20170407 19:41:00 in /builds/slave/rel-m-beta-fx_chcksms-00000000 19:41:00 INFO - Using buildbot properties: 19:41:00 INFO - { 19:41:00 INFO - "project": "", 19:41:00 INFO - "product": "firefox", 19:41:00 INFO - "build_number": 2, 19:41:00 INFO - "taskId": "SbcDole5Q5Km2MuPpecMhg", 19:41:00 INFO - "repository": "", 19:41:00 INFO - "buildername": "release-mozilla-beta-firefox_chcksms", 19:41:00 INFO - "basedir": "/builds/slave/rel-m-beta-fx_chcksms-00000000", 19:41:00 INFO - "buildnumber": 15, 19:41:00 INFO - "platform": null, 19:41:00 INFO - "version": "53.0b10", 19:41:00 INFO - "release_promotion": true, 19:41:00 INFO - "script_repo_revision": "65bd52f60c9f", 19:41:00 INFO - "branch": "mozilla-beta", 19:41:00 INFO - "master": "http://buildbot-master91.bb.releng.usw2.mozilla.com:8001/", 19:41:00 INFO - "revision": "65bd52f60c9f", 19:41:00 INFO - "slavename": "bld-linux64-spot-376", 19:41:00 INFO - "repo_path": "releases/mozilla-beta" 19:41:00 INFO - } 19:41:00 INFO - Overriding version with 53.0b10 19:41:00 INFO - Overriding build_number with 2 19:41:00 INFO - Run as scripts/scripts/release/generate-checksums.py --stage-product firefox --bucket-name-full net-mozaws-prod-delivery-firefox --credentials /builds/release-s3.credentials --tools-repo /tools/checkouts/build-tools 19:41:00 INFO - Dumping config to /builds/slave/rel-m-beta-fx_chcksms-00000000/logs/localconfig.json. 19:41:00 INFO - {'append_to_log': False, 19:41:00 INFO - 'base_work_dir': '/builds/slave/rel-m-beta-fx_chcksms-00000000', 19:41:00 INFO - 'bucket_name_full': 'net-mozaws-prod-delivery-firefox', 19:41:00 INFO - 'build_number': 2, 19:41:00 INFO - 'buildbot_json_path': 'buildprops.json', 19:41:00 INFO - 'credentials': '/builds/release-s3.credentials', 19:41:00 INFO - 'formats': ('sha512', 'sha256'), 19:41:00 INFO - 'includes': ('^.*\\.tar\\.bz2$', 19:41:00 INFO - '^.*\\.tar\\.xz$', 19:41:00 INFO - '^.*\\.dmg$', 19:41:00 INFO - '^.*\\.bundle$', 19:41:00 INFO - '^.*\\.mar$', 19:41:00 INFO - '^.*Setup.*\\.exe$', 19:41:00 INFO - '^.*\\.xpi$', 19:41:00 INFO - '^.*fennec.*\\.apk$'), 19:41:00 INFO - 'log_level': 'info', 19:41:00 INFO - 'log_to_console': True, 19:41:00 INFO - 'opt_config_files': (), 19:41:00 INFO - 'parallelization': 20, 19:41:00 INFO - 'pip_index': True, 19:41:00 INFO - 'stage_product': 'firefox', 19:41:00 INFO - 'tools_repo': '/tools/checkouts/build-tools', 19:41:00 INFO - 'version': u'53.0b10', 19:41:00 INFO - 'virtualenv_modules': ('boto',), 19:41:00 INFO - 'virtualenv_path': 'venv', 19:41:00 INFO - 'volatile_config': {'actions': None, 'add_actions': None, 'no_actions': None}, 19:41:00 INFO - 'work_dir': 'build'} 19:41:00 INFO - [mozharness: 2017-04-08 02:41:00.670085Z] Running create-virtualenv step. 19:41:00 INFO - Running main action method: create_virtualenv 19:41:00 INFO - Creating virtualenv /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv 19:41:00 INFO - mkdir: /builds/slave/rel-m-beta-fx_chcksms-00000000/build 19:41:00 INFO - Running command: ['virtualenv', '--no-site-packages', '--distribute', '/builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv'] in /builds/slave/rel-m-beta-fx_chcksms-00000000/build 19:41:00 INFO - Copy/paste: virtualenv --no-site-packages --distribute /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv 19:41:00 INFO - The --no-site-packages flag is deprecated; it is now the default behavior. 19:41:00 INFO - New python executable in /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/python2.7 19:41:00 INFO - Also creating executable in /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/python 19:41:01 INFO - Installing distribute.............................................................................................................................................................................................done. 19:41:01 INFO - Installing pip...............done. 19:41:01 INFO - Return code: 0 19:41:01 INFO - Getting output from command: ['/builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pip', '--version'] 19:41:01 INFO - Copy/paste: /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pip --version 19:41:01 INFO - Reading from file tmpfile_stdout 19:41:01 INFO - Output received: 19:41:01 INFO - pip 1.1 from /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/lib/python2.7/site-packages/pip-1.1-py2.7.egg (python 2.7) 19:41:01 INFO - Installing boto into virtualenv /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv 19:41:01 INFO - retry: Calling run_command with args: [['/builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pip', 'install', '--timeout', '120', 'boto']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1ee7170>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1d4d420>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1f15fb0>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1eb9100>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1f66c00>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x201bf08>, 'level': 'critical'}], 'cwd': '/builds/slave/rel-m-beta-fx_chcksms-00000000/build', 'env': {'MOZ_AUTOMATION': '1', 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 'TMOUT': '86400', 'LOGNAME': 'cltbld', 'USER': 'cltbld', 'MOZ_OBJDIR': 'obj-firefox', 'BOTO_CONFIG': '/builds/release-s3.credentials', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'DISPLAY': ':2', 'CCACHE_UMASK': '002', 'LANG': 'en_US.UTF-8', 'CCACHE_HASHDIR': '', 'TERM': 'linux', 'SHELL': '/bin/bash', 'MOZ_SIGNING_SERVERS': 'gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120,gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120,gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120,macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'SHLVL': '1', 'G_BROKEN_FILENAMES': '1', 'HISTSIZE': '1000', 'HOME': '/home/cltbld', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'TINDERBOX_OUTPUT': '1', 'MOZ_SIGN_CMD': 'python /tools/checkouts/build-tools/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-fx_chcksms-00000000/signing_cache -t /builds/slave/rel-m-beta-fx_chcksms-00000000/token -n /builds/slave/rel-m-beta-fx_chcksms-00000000/nonce -c /tools/checkouts/build-tools/release/signing/host.cert -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'LC_ALL': 'C', 'TOOLTOOL_HOME': '/builds', '_': '/tools/buildbot/bin/python', 'MAIL': '/var/spool/mail/cltbld', 'HOSTNAME': 'bld-linux64-spot-376.build.releng.usw2.mozilla.com', 'HISTCONTROL': 'ignoredups', 'PWD': '/builds/slave/rel-m-beta-fx_chcksms-00000000', 'PROPERTIES_FILE': '/builds/slave/rel-m-beta-fx_chcksms-00000000/buildprops.json', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_COMPRESS': '1', 'TOOLTOOL_CACHE': '/builds/tooltool_cache'}}, attempt #1 19:41:01 INFO - Running command: ['/builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pip', 'install', '--timeout', '120', 'boto'] in /builds/slave/rel-m-beta-fx_chcksms-00000000/build 19:41:01 INFO - Copy/paste: /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pip install --timeout 120 boto 19:41:01 INFO - Using env: {'BOTO_CONFIG': '/builds/release-s3.credentials', 19:41:01 INFO - 'CCACHE_COMPRESS': '1', 19:41:01 INFO - 'CCACHE_DIR': '/builds/ccache', 19:41:01 INFO - 'CCACHE_HASHDIR': '', 19:41:01 INFO - 'CCACHE_UMASK': '002', 19:41:01 INFO - 'DISPLAY': ':2', 19:41:01 INFO - 'G_BROKEN_FILENAMES': '1', 19:41:01 INFO - 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 19:41:01 INFO - 'HISTCONTROL': 'ignoredups', 19:41:01 INFO - 'HISTSIZE': '1000', 19:41:01 INFO - 'HOME': '/home/cltbld', 19:41:01 INFO - 'HOSTNAME': 'bld-linux64-spot-376.build.releng.usw2.mozilla.com', 19:41:01 INFO - 'LANG': 'en_US.UTF-8', 19:41:01 INFO - 'LC_ALL': 'C', 19:41:01 INFO - 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 19:41:01 INFO - 'LOGNAME': 'cltbld', 19:41:01 INFO - 'MAIL': '/var/spool/mail/cltbld', 19:41:01 INFO - 'MOZ_AUTOMATION': '1', 19:41:01 INFO - 'MOZ_CRASHREPORTER_NO_REPORT': '1', 19:41:01 INFO - 'MOZ_OBJDIR': 'obj-firefox', 19:41:01 INFO - 'MOZ_SIGNING_SERVERS': 'gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120,gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120,gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120,dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120,macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 19:41:01 INFO - 'MOZ_SIGN_CMD': 'python /tools/checkouts/build-tools/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-fx_chcksms-00000000/signing_cache -t /builds/slave/rel-m-beta-fx_chcksms-00000000/token -n /builds/slave/rel-m-beta-fx_chcksms-00000000/nonce -c /tools/checkouts/build-tools/release/signing/host.cert -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 19:41:01 INFO - 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 19:41:01 INFO - 'PROPERTIES_FILE': '/builds/slave/rel-m-beta-fx_chcksms-00000000/buildprops.json', 19:41:01 INFO - 'PWD': '/builds/slave/rel-m-beta-fx_chcksms-00000000', 19:41:01 INFO - 'SHELL': '/bin/bash', 19:41:01 INFO - 'SHLVL': '1', 19:41:01 INFO - 'TERM': 'linux', 19:41:01 INFO - 'TINDERBOX_OUTPUT': '1', 19:41:01 INFO - 'TMOUT': '86400', 19:41:01 INFO - 'TOOLTOOL_CACHE': '/builds/tooltool_cache', 19:41:01 INFO - 'TOOLTOOL_HOME': '/builds', 19:41:01 INFO - 'USER': 'cltbld', 19:41:01 INFO - '_': '/tools/buildbot/bin/python'} 19:41:04 INFO - Ignoring indexes: http://pypi.python.org/simple/ 19:41:04 INFO - Downloading/unpacking boto 19:41:04 INFO - Running setup.py egg_info for package boto 19:41:04 INFO - warning: no files found matching 'boto/mturk/test/*.doctest' 19:41:04 INFO - warning: no files found matching 'boto/mturk/test/.gitignore' 19:41:04 INFO - Installing collected packages: boto 19:41:04 INFO - Running setup.py install for boto 19:41:04 INFO - warning: no files found matching 'boto/mturk/test/*.doctest' 19:41:04 INFO - warning: no files found matching 'boto/mturk/test/.gitignore' 19:41:04 INFO - changing mode of build/scripts-2.7/sdbadmin from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/elbadmin from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/cfadmin from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/s3put from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/fetch_file from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/launch_instance from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/list_instances from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/taskadmin from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/kill_instance from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/bundle_image from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/pyami_sendmail from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/lss3 from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/cq from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/route53 from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/cwutil from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/instance_events from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/asadmin from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/glacier from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/mturk from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/dynamodb_dump from 664 to 775 19:41:04 INFO - changing mode of build/scripts-2.7/dynamodb_load from 664 to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/cfadmin to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/sdbadmin to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/launch_instance to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/route53 to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/list_instances to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/dynamodb_load to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/cq to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/cwutil to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/s3put to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/dynamodb_dump to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/kill_instance to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/bundle_image to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/fetch_file to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/mturk to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/asadmin to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pyami_sendmail to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/lss3 to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/glacier to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/taskadmin to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/elbadmin to 775 19:41:04 INFO - changing mode of /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/instance_events to 775 19:41:04 INFO - Successfully installed boto 19:41:04 INFO - Cleaning up... 19:41:04 INFO - Return code: 0 19:41:04 INFO - Done creating virtualenv /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv. 19:41:04 INFO - Getting output from command: ['/builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pip', 'freeze'] 19:41:04 INFO - Copy/paste: /builds/slave/rel-m-beta-fx_chcksms-00000000/build/venv/bin/pip freeze 19:41:04 INFO - Reading from file tmpfile_stdout 19:41:04 INFO - Current package versions: 19:41:04 INFO - boto == 2.38.0 19:41:04 INFO - distribute == 0.6.24 19:41:04 INFO - wsgiref == 0.1.2 19:41:04 INFO - [mozharness: 2017-04-08 02:41:04.379857Z] Finished create-virtualenv step (success) 19:41:04 INFO - [mozharness: 2017-04-08 02:41:04.380097Z] Running collect-individual-checksums step. 19:41:04 INFO - Running main action method: collect_individual_checksums 19:41:04 INFO - Connecting to S3 19:41:04 INFO - Connecting to bucket net-mozaws-prod-delivery-firefox 19:41:04 INFO - File prefix is: pub/firefox/candidates/53.0b10-candidates/build2/ 19:41:04 INFO - Getting key names from bucket 19:41:13 INFO - Using beet format 19:44:40 INFO - [mozharness: 2017-04-08 02:44:40.081514Z] Finished collect-individual-checksums step (failed) 19:44:40 FATAL - Uncaught exception: Traceback (most recent call last): 19:44:40 FATAL - File "/builds/slave/rel-m-beta-fx_chcksms-00000000/scripts/mozharness/base/script.py", line 2028, in run 19:44:40 FATAL - self.run_action(action) 19:44:40 FATAL - File "/builds/slave/rel-m-beta-fx_chcksms-00000000/scripts/mozharness/base/script.py", line 1967, in run_action 19:44:40 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 19:44:40 FATAL - File "/builds/slave/rel-m-beta-fx_chcksms-00000000/scripts/mozharness/base/script.py", line 1907, in _possibly_run_method 19:44:40 FATAL - return getattr(self, method_name)() 19:44:40 FATAL - File "scripts/scripts/release/generate-checksums.py", line 216, in collect_individual_checksums 19:44:40 FATAL - pool.map(worker, find_checksums_files()) 19:44:40 FATAL - File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 227, in map 19:44:40 FATAL - return self.map_async(func, iterable, chunksize).get() 19:44:40 FATAL - File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 528, in get 19:44:40 FATAL - raise self._value 19:44:40 FATAL - BotoServerError: BotoServerError: 500 Internal Server Error 19:44:40 FATAL - 19:44:40 FATAL - Running post_fatal callback... 19:44:40 FATAL - Exiting -1 19:44:40 INFO - Running post-run listener: copy_logs_to_upload_dir 19:44:40 INFO - Copying logs to upload dir... 19:44:40 INFO - mkdir: /builds/slave/rel-m-beta-fx_chcksms-00000000/build/upload/logs program finished with exit code 255 elapsedTime=219.649169 ========= master_lag: 0.01 ========= ========= Finished '/tools/buildbot/bin/python scripts/scripts/release/generate-checksums.py ...' failed (results: 2, elapsed: 3 mins, 39 secs) (at 2017-04-07 19:44:40.161310) ========= ========= Started no change (results: 0, elapsed: 0 secs) (at 2017-04-07 19:44:40.162698) ========= bash -c 'for file in `ls -1`; do cat $file; done' in dir /builds/slave/rel-m-beta-fx_chcksms-00000000/properties (timeout 1200 secs) watching logfiles {} argv: ['bash', '-c', 'for file in `ls -1`; do cat $file; done'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-376.build.releng.usw2.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-fx_chcksms-00000000/properties SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.005049 ========= master_lag: 0.04 ========= ========= Finished no change (results: 0, elapsed: 0 secs) (at 2017-04-07 19:44:40.209864) ========= ========= Started reboot skipped (results: 3, elapsed: 0 secs) (at 2017-04-07 19:44:40.210167) ========= ========= Finished reboot skipped (results: 3, elapsed: 0 secs) (at 2017-04-07 19:44:40.210408) ========= ========= Total master_lag: 0.20 =========