2.FastRunner定时任务Celery+RabbitMQ

news2024/11/23 18:40:33

注意：celery版本和Python冲突问题

不能用高版本Python 用3.5以下，因为项目的celery用的django-celery == 3.2.2

python3.7 async关键字

冲突版本 celery3.x
方案一： celery3.x+python3.6
方案二： celery4.x+python3.7

解决celery执行时报错：KeyError: 'async' 报错原因：Python3.6及以上版本和celebry4.0以上版本中async关键字发生冲突。

原因分析：
1.这是因为在 python 3.7 中将 async 作为了关键字，所以当 py 文件中出现类似 from . import async, base 这类不符合python语法的语句时，Python会报错；
解决方法：在 celery 官方的提议下，建议将 kombu下的async.py 文件的文件名改成 asynchronous；然后把引用和这个文件的所有文件的里面的async改为asynchronous；
2. 但是再修改后会报错:No module named ‘kombu.asynchronous’,搜索到的方法是回退版本,然后再次升级到最新版本,不再出现报错(有些人进回退版本就好):
pip uninstall celery && pip install celery==3.1.25;
pip install celery;不过对我都没用,因为发现回退版本后会重置async的修改,进行第一步的报错
3. 之后我从kombu下手,删除原有的kombu,并安装最新的版本后,问题解决:
pip uninstall kombu && pip install kombu

django与 python版本

Django 1.11版本兼容Python 2.7、3.4、3.5、3.6、3.7（added in 1.11.17）# 1.11.5 python<=3.6

Django 2.0版本兼容Python 3.4、3.5、3.6、3.7

Django 2.1、2.2版本兼容Python 3.5、3.6、3.7

Django 3.0版本兼容Python 3.6、3.7、3.8

djangorestframework与django python版本

djangorestframework 3.6.4 (python2.7 django1.11.5)
- Python (2.7, 3.2, 3.3, 3.4, 3.5, 3.6) # drf <= 3.8 support python2.7
- Django (1.8, 1.9, 1.10, 1.11)

djangorestframework 3.10.0
- Python (3.5, 3.6, 3.7) # drf >= 3.9 support python3.7
- Django (1.11, 2.0, 2.1, 2.2)

djangorestframework 3.11.2 (python3.7 django1.11.17 drf3.11.2)
- Python (3.5, 3.6, 3.7, 3.8)
- Django (1.11, 2.0, 2.1, 2.2, 3.0) # the last version which support django 1.11

djangorestframework 3.12(latest version 3.12.4 2021-03)
- Python (3.5, 3.6, 3.7, 3.8, 3.9)
- Django (2.2, 3.0, 3.1) # django >= 2.0

1.启动定时任务

1.安装rabbmitMQ中间件，并配置setting中的BROKER_URL（默认一般不用修改）

登陆RabbitMQ后台

Centos7

# 注意防火墙是否开放端口
# 防火墙相关操作 https://www.jb51.net/article/147012.htm
http://ip:15672

Windows10

http://localhost:15672

2. setting.py

djcelery.setup_loader()
CELERY_ENABLE_UTC = True
CELERY_TIMEZONE = 'Asia/Shanghai'
# BROKER_URL = 'amqp://username:password@IP:5672//'
BROKER_URL = 'amqp://guest:guest@127.0.0.1:5672//'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

CELERY_TASK_RESULT_EXPIRES = 7200
CELERYD_CONCURRENCY = 1 if DEBUG else 5
CELERYD_MAX_TASKS_PER_CHILD = 40

3.启动定时服务

1.启动Celery Beat(相当于发送消息给RabbitMQ)

# 使用默认的celery.py启动
python3 manage.py celery beat -l info > ./logs/beat.log 2>&1

# 指定的celery.py启动,其中的 -A是指定文件

celery -A FasterRunner.celery beat -l info > ~/logs/beat.log 2>&1

2.启动Celery work(处理RabbitMQ中的消息)

# 使用默认的celery.py启动
celery multi start w1 -A FasterRunner -l info --logfile=./logs/worker.log

# 指定的celery.py启动,其中的 -A是指定文件
celery multi start w1 -A FasterRunner.celery -l info --logfile=./logs/worker.log

celery -A FasterRunner worker -l info 启动异步worker
python manage.py celery beat -l info 启动beat监听定时任务

4.调试定时任务

发送消息日志

# 发送消息日志
(fastrunner) [root@izwz9awyk38uq20rb3czmnz FasterRunner]# tailf logs/beat.log 
[2019-06-24 18:09:00,000: INFO/MainProcess] Scheduler: Sending due task 每分钟执行一次 (fastrunner.tasks.schedule_debug_suite)
[2019-06-24 18:10:00,000: INFO/MainProcess] Scheduler: Sending due task 每分钟执行一次 (fastrunner.tasks.schedule_debug_suite)

处理消息日志

# 处理消息日志
(fastrunner) [root@izwz9awyk38uq20rb3czmnz FasterRunner]# tailf ./logs/worker.log
[2019-06-24 18:09:00,172: WARNING/Worker-5] Ran 1 test in 0.138s
[2019-06-24 18:09:00,173: WARNING/Worker-5] OK
[2019-06-24 18:09:00,173: DEBUG/Worker-5] 
================== Variables & Output ==================
Type   | Variable         :  Value
------ | ---------------- :  ---------------------------

--------------------------------------------------------

[2019-06-24 18:09:00,199: INFO/MainProcess] Task fastrunner.tasks.schedule_debug_suite[adda0f06-ed65-470c-a7db-7042707f3fd6] succeeded in 0.19341683201491833s: None
[2019-06-24 18:10:00,004: INFO/MainProcess] Received task: fastrunner.tasks.schedule_debug_suite[5eecf0ce-afd5-4b2a-8d60-55a184f093bf]
[2019-06-24 18:10:00,024: WARNING/Worker-1] 111
[2019-06-24 18:10:00,024: WARNING/Worker-1] 111
[2019-06-24 18:10:00,027: INFO/Worker-1] Start to run testcase: 测试用例_登陆
[2019-06-24 18:10:00,028: WARNING/Worker-1] 登陆
[2019-06-24 18:10:00,028: DEBUG/Worker-1] call hook: ${setup_hook_prepare_kwargs($request)}
[2019-06-24 18:10:00,028: INFO/Worker-1] POST /api/user/login/
[2019-06-24 18:10:00,029: DEBUG/Worker-1] request kwargs(raw): {'headers': {'Content-Type': 'application/json'}, 'verify': False, 'json': {'username': 'fastrunner', 'password': 'fastrunner'}}
[2019-06-24 18:10:00,029: DEBUG/Worker-1] processed request:
> POST http://119.23.50.82:8000/api/user/login/
> kwargs: {'headers': {'Content-Type': 'application/json'}, 'verify': False, 'json': {'username': 'fastrunner', 'password': 'fastrunner'}, 'timeout': 120}
[2019-06-24 18:10:00,160: DEBUG/Worker-1] 
================== request details ==================
url              : 'http://ip:8000/api/user/login/'
method           : 'POST'
headers          : {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '52'}
start_timestamp  : 1561371000.029365
verify           : False
json             : {'username': 'fastrunner', 'password': 'fastrunner'}
body             : b'{"username": "fastrunner", "password": "fastrunner"}'

[2019-06-24 18:10:00,161: DEBUG/Worker-1] 
================== response details ==================
status_code      : 200
headers          : {'Server': 'nginx', 'Date': 'Mon, 24 Jun 2019 10:10:00 GMT', 'Content-Type': 'application/json', 'Content-Length': '115', 'Connection': 'keep-alive', 'Vary': 'Accept, Origin', 'Allow': 'POST, OPTIONS', 'X-Frame-Options': 'SAMEORIGIN'}
content_size     : 115
response_time_ms : 131.36
elapsed_ms       : 128.769
encoding         : None
content          : b'{"code":"0001","success":true,"msg":"login success","token":"22b3bf6afd31435e2bdce609713ed7b9","user":"fastrunner"}'
content_type     : 'application/json'
ok               : True
url              : 'http://ip:8000/api/user/login/'
reason           : 'OK'
cookies          : {}
text             : '{"code":"0001","success":true,"msg":"login success","token":"22b3bf6afd31435e2bdce609713ed7b9","user":"fastrunner"}'
json             : {'code': '0001', 'success': True, 'msg': 'login success', 'token': '22b3bf6afd31435e2bdce609713ed7b9', 'user': 'fastrunner'}