New version

This commit is contained in:
2020-11-04 00:23:14 +03:00
parent 8c374d93bc
commit d2d31dfeff
157 changed files with 2650 additions and 1686 deletions

File diff suppressed because one or more lines are too long

Binary file not shown.

View File

@@ -1,138 +1,138 @@
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class="no-js"> <head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><title>Долгие сообщения в RabbitMQ</title><meta name="description" content><meta name="viewport" content="width=device-width"><link rel="stylesheet" href="../../theme/css/normalize.css"><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel="stylesheet"><link rel="stylesheet" href="../../theme/css/font-awesome.min.css"><link rel="stylesheet" href="../../theme/css/main.css"><link rel="stylesheet" href="../../theme/css/blog.css"><link rel="stylesheet" href="../../theme/css/github.css"><link href="https://likemath.ru/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Блог 529 Atom Feed"><link href="https://likemath.ru/feeds/all.rss.xml" type="application/rss+xml" rel="alternate" title="Блог 529 RSS Feed"><script src="../../theme/js/vendor/modernizr-2.6.2.min.js"></script></head><body><!--[if lt IE 7]>
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class=no-js> <!--<![endif]--> <head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><title>Долгие сообщения в RabbitMQ</title><meta name=description content><meta name=viewport content="width=device-width"><link rel=stylesheet href=../../theme/css/normalize.css><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel=stylesheet><link rel=stylesheet href=../../theme/css/font-awesome.min.css><link rel=stylesheet href=../../theme/css/main.css><link rel=stylesheet href=../../theme/css/blog.css><link rel=stylesheet href=../../theme/css/github.css><link href=https://likemath.ru/feeds/all.atom.xml type=application/atom+xml rel=alternate title="Блог 529 Atom Feed"><link href=https://likemath.ru/feeds/all.rss.xml type=application/rss+xml rel=alternate title="Блог 529 RSS Feed"><script src=../../theme/js/vendor/modernizr-2.6.2.min.js></script></head> <body> <!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]--><div id="wrapper"><header id="sidebar" class="side-shadow"><hgroup id="site-header"><a id="site-title" href="../.."><h2><i class="icon-pencil"></i> Блог 529</h2></a><p id="site-desc"> Project Euler и остальное </p></hgroup><nav><ul id="nav-links"><li><a href="../../">Главная</a></li><li><a href="../../pages/projects.html">Мои проекты</a></li><li><a href="../../pages/about.html">Об авторе</a></li><li><a href="../../feeds/feed.atom.xml">Atom feed</a></li></ul></nav><footer id="site-info"><p> Powered by Pelican. </p></footer></header><div id="post-container"><ol id="post-list"><li><article class="post-entry"><header class="entry-header"><time class="post-time" datetime="2018-10-01T12:40:00+03:00" pubdate> Пн 01 Октябрь 2018 </time><a href="../../posts/dolgie-soobshcheniia-v-rabbitmq/" rel="bookmark"><h1>Долгие сообщения в&nbsp;RabbitMQ</h1></a></header><section class="post-content"><p>Предположим, что у вас появилось желание перекодировать фильмы на вашем медиасервере, и вы решили использовать production-ready решение для хранения заданий. Вы взяли RabbitMQ для управления очередями сообщений и Python для их обработки. Но почему-то сообщения обрабатываются нестабильно, клиент падает без всяких видимых причин. Попробуем понять почему такое может&nbsp;быть.</p><p>Возьмём готовый код из официального <a href="https://www.rabbitmq.com/tutorials/tutorial-one-python.html">туториала</a> RabbitMQ и немного его модифицируем, чтобы он обрабатывал сообщения за различное&nbsp;время.</p><h3>Producer</h3><div class="highlight"><pre><span class="code-line"><span></span><span class="ch">#!/usr/bin/python</span></span>
<span class="code-line"><span class="c1"># -*- coding: utf-8 -*-</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="n">__author__</span> <span class="o">=</span> <span class="s2">&quot;Aleksey Lobanov&quot;</span></span>
<span class="code-line"><span class="n">__license__</span> <span class="o">=</span> <span class="s2">&quot;MIT&quot;</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">sys</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">pika</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">connection</span> <span class="o">=</span> <span class="n">pika</span><span class="o">.</span><span class="n">BlockingConnection</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">pika</span><span class="o">.</span><span class="n">ConnectionParameters</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s1">&#39;localhost&#39;</span><span class="p">)</span></span>
<span class="code-line"> <span class="p">)</span></span>
<span class="code-line"> <span class="n">channel</span> <span class="o">=</span> <span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span></span>
<span class="code-line"></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">queue_declare</span><span class="p">(</span><span class="n">queue</span><span class="o">=</span><span class="s1">&#39;demo.hello&#39;</span><span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="c1"># чтение первого аргумента командной строки, если он есть</span></span>
<span class="code-line"> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">delay_to_send</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span></span>
<span class="code-line"> <span class="k">else</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">delay_to_send</span> <span class="o">=</span> <span class="mi">3</span></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">basic_publish</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">exchange</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">routing_key</span><span class="o">=</span><span class="s1">&#39;demo.hello&#39;</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">body</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">delay_to_send</span><span class="p">)</span></span>
<span class="code-line"> <span class="p">)</span></span>
<span class="code-line"> <span class="k">print</span><span class="p">(</span><span class="s2">&quot; [x] Sent &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">delay_to_send</span><span class="p">))</span></span>
<span class="code-line"> <span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></span>
</pre></div><h3>Consumer</h3><div class="highlight"><pre><span class="code-line"><span></span><span class="ch">#!/usr/bin/python</span></span>
<span class="code-line"><span class="c1"># -*- coding: utf-8 -*-</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="n">__author__</span> <span class="o">=</span> <span class="s2">&quot;Aleksey Lobanov&quot;</span></span>
<span class="code-line"><span class="n">__license__</span> <span class="o">=</span> <span class="s2">&quot;MIT&quot;</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">sys</span></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">time</span></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">math</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">pika</span></span>
<span class="code-line"></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">def</span> <span class="nf">callback</span><span class="p">(</span><span class="n">ch</span><span class="p">,</span> <span class="n">method</span><span class="p">,</span> <span class="n">properties</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span></span>
<span class="code-line"> <span class="k">print</span><span class="p">(</span><span class="s2">&quot; [x] Received </span><span class="si">%r</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="n">body</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">delay</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">body</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">begin_at</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span></span>
<span class="code-line"> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">delay</span><span class="p">)</span></span>
<span class="code-line"> <span class="k">print</span><span class="p">(</span><span class="s2">&quot; [x] Finished {}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">body</span><span class="p">))</span></span>
<span class="code-line"></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">connection</span> <span class="o">=</span> <span class="n">pika</span><span class="o">.</span><span class="n">BlockingConnection</span><span class="p">(</span><span class="n">pika</span><span class="o">.</span><span class="n">ConnectionParameters</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s1">&#39;localhost&#39;</span><span class="p">))</span></span>
<span class="code-line"> <span class="n">channel</span> <span class="o">=</span> <span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">queue_declare</span><span class="p">(</span><span class="n">queue</span><span class="o">=</span><span class="s1">&#39;demo.hello&#39;</span><span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">basic_consume</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">callback</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">queue</span><span class="o">=</span><span class="s1">&#39;demo.hello&#39;</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">no_ack</span><span class="o">=</span><span class="bp">True</span></span>
<span class="code-line"> <span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">print</span><span class="p">(</span><span class="s1">&#39; [*] Waiting for messages. To exit press CTRL+C&#39;</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">start_consuming</span><span class="p">()</span></span>
</pre></div><p>Пока мы посылаем маленькие числа, всё будет хорошо работать. Но если послать какое-то большое (в моём случае достаточно 200), то consumer потеряет соединение с сервером, будет ошибка. Скорее всего это будет <code>pika.exceptions.ConnectionClosed: (-1, 'EOF')</code> или <code>Socket Error 104</code> (<a href="https://github.com/pika/pika/issues/753">тут</a> есть обсуждение на GitHub библиотеки, но установка <code>prefetch_count=1</code> тоже не поможет). Эта проблема актуальна для обоих веток&nbsp;Python.</p><p>Реальная причина в том, что при обработке сообщения не происходит необходимого взаимодействия с RabbitMQ, не отправляются hearbeats, а без них сервер считает, что этот клиент погиб окончательно. Нужно отметить, что переход на другой тип соединения не помогает. Например использование <a href="https://pika.readthedocs.io/en/stable/examples/twisted_example.html">примера</a> на Twisted из официальной документации ничего не&nbsp;изменит.</p><p>У этой проблемы есть много решений. Будем считать, что сообщение, которое мы обрабатываем не разделяется на подзадачи и рассмотрим некоторые из&nbsp;них:</p><ol><li>Отключить hearbeats/увеличить их интервал так, чтобы самprefetch_countая долгая обработка сообщения вела к потере не более, чем одного. Это самое простое решение, но в таком случае мы теряем в надёжности. Внешнем сервисам мониторинга будет сложнее понять, обрабатываются ли сейчас сообщения или уже нет. И чем больше интервал, тем серьёзнее проблема. <strong>Не&nbsp;подходит</strong></li><li>Разбить обработку сообщения на несколько этапов. Часто это хорошее решение, но в предположении, что данная обработка не разбивается на более мелкие тоже <strong>не подходит</strong>. </li><li>Использовать <code>connection.sleep</code> вместо <code>time.sleep</code>, а также регулярный вызов <code>BlockingConnection.process_data_events</code>. Оба эти решения помогают, но их использование плохо по многим причинам. Во-первых это явное протекании абстракции, когда код для обработки сообщения вынужден работать с очередью. Во-вторых не всегда можно гарантировать, что эти функции вызываются достаточно часто, а это главная проблема. <strong>Не&nbsp;подходит</strong></li><li>Выделить отдельный процесс под <code>pika</code>. Вероятно, единственно универсальное решение. Если исходный код был правильно написан, то адаптация будет простой. Но у этого решения есть минусы, обязательные при использовании нескольких потоков/процессов. Также сама библиотека <code>pika</code> не является&nbsp;потокобезопасной.</li></ol><p>Возможное решение будет заключаться в вынесение обработки данных в отдельный&nbsp;метод:</p><div class="highlight"><pre><span class="code-line"><span></span><span class="k">def</span> <span class="nf">real_work</span><span class="p">(</span><span class="n">body</span><span class="p">):</span></span>
<span class="code-line"> <span class="n">delay</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">body</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">begin_at</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span></span>
<span class="code-line"> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">delay</span><span class="p">)</span></span>
</pre></div><p>и небольшой доработке&nbsp;callback-метода:</p><div class="highlight"><pre><span class="code-line"><span></span><span class="c1"># Конструктор, при запуске процесс запустит</span></span>
<span class="code-line"><span class="c1"># функцию с указанными параметрами</span></span>
<span class="code-line"><span class="n">work_process</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">target</span><span class="o">=</span><span class="n">real_work</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="p">)</span></span>
<span class="code-line"><span class="p">)</span></span>
<span class="code-line"><span class="n">work_process</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="c1"># процесс нужно явно запустить</span></span>
<span class="code-line"><span class="k">while</span> <span class="bp">True</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">ch</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span></span>
<span class="code-line"> <span class="c1"># пока поток работает, используем функцию pika</span></span>
<span class="code-line"> <span class="c1"># для обработки необходимых сообщений</span></span>
<span class="code-line"> <span class="k">if</span> <span class="ow">not</span> <span class="n">work_process</span><span class="o">.</span><span class="n">is_alive</span><span class="p">():</span></span>
<span class="code-line"> <span class="k">break</span></span>
</pre></div><p>Тогда готовый код будет выглядеть&nbsp;так:</p><div class="highlight"><pre><span class="code-line"><span></span><span class="ch">#!/usr/bin/python</span></span>
<span class="code-line"><span class="c1"># -*- coding: utf-8 -*-</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="n">__author__</span> <span class="o">=</span> <span class="s2">&quot;Aleksey Lobanov&quot;</span></span>
<span class="code-line"><span class="n">__license__</span> <span class="o">=</span> <span class="s2">&quot;MIT&quot;</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">sys</span></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">time</span></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">math</span></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">multiprocessing</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">pika</span></span>
<span class="code-line"></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">def</span> <span class="nf">real_work</span><span class="p">(</span><span class="n">body</span><span class="p">):</span></span>
<span class="code-line"> <span class="n">delay</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">body</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">begin_at</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span></span>
<span class="code-line"> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">delay</span><span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">def</span> <span class="nf">callback</span><span class="p">(</span><span class="n">ch</span><span class="p">,</span> <span class="n">method</span><span class="p">,</span> <span class="n">properties</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span></span>
<span class="code-line"> <span class="k">print</span><span class="p">(</span><span class="s2">&quot; [x] Received </span><span class="si">%r</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="n">body</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">work_process</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">target</span><span class="o">=</span><span class="n">real_work</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="p">)</span></span>
<span class="code-line"> <span class="p">)</span></span>
<span class="code-line"> <span class="n">work_process</span><span class="o">.</span><span class="n">start</span><span class="p">()</span></span>
<span class="code-line"> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">ch</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span></span>
<span class="code-line"> <span class="k">if</span> <span class="ow">not</span> <span class="n">work_process</span><span class="o">.</span><span class="n">is_alive</span><span class="p">():</span></span>
<span class="code-line"> <span class="k">break</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">print</span><span class="p">(</span><span class="s2">&quot; [x] Finished &quot;</span> <span class="o">+</span> <span class="n">body</span><span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">connection</span> <span class="o">=</span> <span class="n">pika</span><span class="o">.</span><span class="n">BlockingConnection</span><span class="p">(</span><span class="n">pika</span><span class="o">.</span><span class="n">ConnectionParameters</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s1">&#39;localhost&#39;</span><span class="p">))</span></span>
<span class="code-line"> <span class="n">channel</span> <span class="o">=</span> <span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">queue_declare</span><span class="p">(</span><span class="n">queue</span><span class="o">=</span><span class="s1">&#39;demo.hello&#39;</span><span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">basic_consume</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">callback</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">queue</span><span class="o">=</span><span class="s1">&#39;demo.hello&#39;</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">no_ack</span><span class="o">=</span><span class="bp">True</span></span>
<span class="code-line"> <span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">print</span><span class="p">(</span><span class="s1">&#39; [*] Waiting for messages. To exit press CTRL+C&#39;</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">channel</span><span class="o">.</span><span class="n">start_consuming</span><span class="p">()</span></span>
</pre></div></section><hr><aside class="post-meta"><p>Категория: <a href="../../category/python.html">Python</a></p><p>Теги: <a href="../../tag/python.html">Python</a>, <a href="../../tag/rabbitmq.html">RabbitMQ</a>, </p></aside><hr></article></li></ol></div></div><script type="text/javascript">
<![endif]--> <div id=wrapper> <header id=sidebar class=side-shadow> <hgroup id=site-header> <a id=site-title href=../..><h2><i class=icon-pencil></i> Блог 529</h2></a> <p id=site-desc> Project Euler и остальное </p> </hgroup> <nav> <ul id=nav-links> <li><a href=../../ >Главная</a></li> <li><a href=../../pages/projects.html>Мои проекты</a></li> <li><a href=../../pages/about.html>Об авторе</a></li> <li><a href=../../feeds/feed.atom.xml>Atom feed</a></li> </ul> </nav> <footer id=site-info> <p> Powered by Pelican. </p> </footer></header> <div id=post-container> <ol id=post-list> <li> <article class=post-entry> <header class=entry-header> <time class=post-time datetime=2018-10-01T12:40:00+03:00 pubdate> Пн 01 октября 2018 </time> <a href=../../posts/dolgie-soobshcheniia-v-rabbitmq/ rel=bookmark><h1>Долгие сообщения в&nbsp;RabbitMQ</h1></a> </header> <section class=post-content> <p>Предположим, что у вас появилось желание перекодировать фильмы на вашем медиасервере, и вы решили использовать production-ready решение для хранения заданий. Вы взяли RabbitMQ для управления очередями сообщений и Python для их обработки. Но почему-то сообщения обрабатываются нестабильно, клиент падает без всяких видимых причин. Попробуем понять почему такое может&nbsp;быть.</p> <p>Возьмём готовый код из официального <a href=https://www.rabbitmq.com/tutorials/tutorial-one-python.html>туториала</a> RabbitMQ и немного его модифицируем, чтобы он обрабатывал сообщения за различное&nbsp;время.</p> <h3>Producer</h3> <div class=highlight><pre><span class=code-line><span></span><span class=ch>#!/usr/bin/python</span></span>
<span class=code-line><span class=c1># -*- coding: utf-8 -*-</span></span>
<span class=code-line></span>
<span class=code-line><span class=n>__author__</span> <span class=o>=</span> <span class=s2>&quot;Aleksey Lobanov&quot;</span></span>
<span class=code-line><span class=n>__license__</span> <span class=o>=</span> <span class=s2>&quot;MIT&quot;</span></span>
<span class=code-line></span>
<span class=code-line><span class=kn>import</span> <span class=nn>sys</span></span>
<span class=code-line></span>
<span class=code-line><span class=kn>import</span> <span class=nn>pika</span></span>
<span class=code-line></span>
<span class=code-line><span class=k>if</span> <span class=vm>__name__</span> <span class=o>==</span> <span class=s2>&quot;__main__&quot;</span><span class=p>:</span></span>
<span class=code-line> <span class=n>connection</span> <span class=o>=</span> <span class=n>pika</span><span class=o>.</span><span class=n>BlockingConnection</span><span class=p>(</span></span>
<span class=code-line> <span class=n>pika</span><span class=o>.</span><span class=n>ConnectionParameters</span><span class=p>(</span><span class=n>host</span><span class=o>=</span><span class=s1>&#39;localhost&#39;</span><span class=p>)</span></span>
<span class=code-line> <span class=p>)</span></span>
<span class=code-line> <span class=n>channel</span> <span class=o>=</span> <span class=n>connection</span><span class=o>.</span><span class=n>channel</span><span class=p>()</span></span>
<span class=code-line></span>
<span class=code-line></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>queue_declare</span><span class=p>(</span><span class=n>queue</span><span class=o>=</span><span class=s1>&#39;demo.hello&#39;</span><span class=p>)</span></span>
<span class=code-line></span>
<span class=code-line> <span class=c1># чтение первого аргумента командной строки, если он есть</span></span>
<span class=code-line> <span class=k>if</span> <span class=nb>len</span><span class=p>(</span><span class=n>sys</span><span class=o>.</span><span class=n>argv</span><span class=p>)</span> <span class=o>&gt;</span> <span class=mi>1</span><span class=p>:</span></span>
<span class=code-line> <span class=n>delay_to_send</span> <span class=o>=</span> <span class=nb>int</span><span class=p>(</span><span class=n>sys</span><span class=o>.</span><span class=n>argv</span><span class=p>[</span><span class=mi>1</span><span class=p>])</span></span>
<span class=code-line> <span class=k>else</span><span class=p>:</span></span>
<span class=code-line> <span class=n>delay_to_send</span> <span class=o>=</span> <span class=mi>3</span></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>basic_publish</span><span class=p>(</span></span>
<span class=code-line> <span class=n>exchange</span><span class=o>=</span><span class=s1>&#39;&#39;</span><span class=p>,</span></span>
<span class=code-line> <span class=n>routing_key</span><span class=o>=</span><span class=s1>&#39;demo.hello&#39;</span><span class=p>,</span></span>
<span class=code-line> <span class=n>body</span><span class=o>=</span><span class=nb>str</span><span class=p>(</span><span class=n>delay_to_send</span><span class=p>)</span></span>
<span class=code-line> <span class=p>)</span></span>
<span class=code-line> <span class=k>print</span><span class=p>(</span><span class=s2>&quot; [x] Sent &quot;</span> <span class=o>+</span> <span class=nb>str</span><span class=p>(</span><span class=n>delay_to_send</span><span class=p>))</span></span>
<span class=code-line> <span class=n>connection</span><span class=o>.</span><span class=n>close</span><span class=p>()</span></span>
</pre></div> <h3>Consumer</h3> <div class=highlight><pre><span class=code-line><span></span><span class=ch>#!/usr/bin/python</span></span>
<span class=code-line><span class=c1># -*- coding: utf-8 -*-</span></span>
<span class=code-line></span>
<span class=code-line><span class=n>__author__</span> <span class=o>=</span> <span class=s2>&quot;Aleksey Lobanov&quot;</span></span>
<span class=code-line><span class=n>__license__</span> <span class=o>=</span> <span class=s2>&quot;MIT&quot;</span></span>
<span class=code-line></span>
<span class=code-line><span class=kn>import</span> <span class=nn>sys</span></span>
<span class=code-line><span class=kn>import</span> <span class=nn>time</span></span>
<span class=code-line><span class=kn>import</span> <span class=nn>math</span></span>
<span class=code-line></span>
<span class=code-line><span class=kn>import</span> <span class=nn>pika</span></span>
<span class=code-line></span>
<span class=code-line></span>
<span class=code-line><span class=k>def</span> <span class=nf>callback</span><span class=p>(</span><span class=n>ch</span><span class=p>,</span> <span class=n>method</span><span class=p>,</span> <span class=n>properties</span><span class=p>,</span> <span class=n>body</span><span class=p>):</span></span>
<span class=code-line> <span class=k>print</span><span class=p>(</span><span class=s2>&quot; [x] Received </span><span class=si>%r</span><span class=s2>&quot;</span> <span class=o>%</span> <span class=n>body</span><span class=p>)</span></span>
<span class=code-line> <span class=n>delay</span> <span class=o>=</span> <span class=nb>int</span><span class=p>(</span><span class=n>body</span><span class=p>)</span></span>
<span class=code-line> <span class=n>begin_at</span> <span class=o>=</span> <span class=n>time</span><span class=o>.</span><span class=n>time</span><span class=p>()</span></span>
<span class=code-line> <span class=n>time</span><span class=o>.</span><span class=n>sleep</span><span class=p>(</span><span class=n>delay</span><span class=p>)</span></span>
<span class=code-line> <span class=k>print</span><span class=p>(</span><span class=s2>&quot; [x] Finished {}&quot;</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>body</span><span class=p>))</span></span>
<span class=code-line></span>
<span class=code-line></span>
<span class=code-line><span class=k>if</span> <span class=vm>__name__</span> <span class=o>==</span> <span class=s2>&quot;__main__&quot;</span><span class=p>:</span></span>
<span class=code-line> <span class=n>connection</span> <span class=o>=</span> <span class=n>pika</span><span class=o>.</span><span class=n>BlockingConnection</span><span class=p>(</span><span class=n>pika</span><span class=o>.</span><span class=n>ConnectionParameters</span><span class=p>(</span><span class=n>host</span><span class=o>=</span><span class=s1>&#39;localhost&#39;</span><span class=p>))</span></span>
<span class=code-line> <span class=n>channel</span> <span class=o>=</span> <span class=n>connection</span><span class=o>.</span><span class=n>channel</span><span class=p>()</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>queue_declare</span><span class=p>(</span><span class=n>queue</span><span class=o>=</span><span class=s1>&#39;demo.hello&#39;</span><span class=p>)</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>basic_consume</span><span class=p>(</span></span>
<span class=code-line> <span class=n>callback</span><span class=p>,</span></span>
<span class=code-line> <span class=n>queue</span><span class=o>=</span><span class=s1>&#39;demo.hello&#39;</span><span class=p>,</span></span>
<span class=code-line> <span class=n>no_ack</span><span class=o>=</span><span class=bp>True</span></span>
<span class=code-line> <span class=p>)</span></span>
<span class=code-line></span>
<span class=code-line> <span class=k>print</span><span class=p>(</span><span class=s1>&#39; [*] Waiting for messages. To exit press CTRL+C&#39;</span><span class=p>)</span></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>start_consuming</span><span class=p>()</span></span>
</pre></div> <p>Пока мы посылаем маленькие числа, всё будет хорошо работать. Но если послать какое-то большое (в моём случае достаточно 200), то consumer потеряет соединение с сервером, будет ошибка. Скорее всего это будет <code>pika.exceptions.ConnectionClosed: (-1, 'EOF')</code> или <code>Socket Error 104</code> (<a href=https://github.com/pika/pika/issues/753>тут</a> есть обсуждение на GitHub библиотеки, но установка <code>prefetch_count=1</code> тоже не поможет). Эта проблема актуальна для обоих веток&nbsp;Python.</p> <p>Реальная причина в том, что при обработке сообщения не происходит необходимого взаимодействия с RabbitMQ, не отправляются hearbeats, а без них сервер считает, что этот клиент погиб окончательно. Нужно отметить, что переход на другой тип соединения не помогает. Например использование <a href=https://pika.readthedocs.io/en/stable/examples/twisted_example.html>примера</a> на Twisted из официальной документации ничего не&nbsp;изменит.</p> <p>У этой проблемы есть много решений. Будем считать, что сообщение, которое мы обрабатываем не разделяется на подзадачи и рассмотрим некоторые из&nbsp;них:</p> <ol> <li>Отключить hearbeats/увеличить их интервал так, чтобы самprefetch_countая долгая обработка сообщения вела к потере не более, чем одного. Это самое простое решение, но в таком случае мы теряем в надёжности. Внешнем сервисам мониторинга будет сложнее понять, обрабатываются ли сейчас сообщения или уже нет. И чем больше интервал, тем серьёзнее проблема. <strong>Не&nbsp;подходит</strong></li> <li>Разбить обработку сообщения на несколько этапов. Часто это хорошее решение, но в предположении, что данная обработка не разбивается на более мелкие тоже <strong>не подходит</strong>. </li> <li>Использовать <code>connection.sleep</code> вместо <code>time.sleep</code>, а также регулярный вызов <code>BlockingConnection.process_data_events</code>. Оба эти решения помогают, но их использование плохо по многим причинам. Во-первых это явное протекании абстракции, когда код для обработки сообщения вынужден работать с очередью. Во-вторых не всегда можно гарантировать, что эти функции вызываются достаточно часто, а это главная проблема. <strong>Не&nbsp;подходит</strong></li> <li>Выделить отдельный процесс под <code>pika</code>. Вероятно, единственно универсальное решение. Если исходный код был правильно написан, то адаптация будет простой. Но у этого решения есть минусы, обязательные при использовании нескольких потоков/процессов. Также сама библиотека <code>pika</code> не является&nbsp;потокобезопасной.</li> </ol> <p>Возможное решение будет заключаться в вынесение обработки данных в отдельный&nbsp;метод:</p> <div class=highlight><pre><span class=code-line><span></span><span class=k>def</span> <span class=nf>real_work</span><span class=p>(</span><span class=n>body</span><span class=p>):</span></span>
<span class=code-line> <span class=n>delay</span> <span class=o>=</span> <span class=nb>int</span><span class=p>(</span><span class=n>body</span><span class=p>)</span></span>
<span class=code-line> <span class=n>begin_at</span> <span class=o>=</span> <span class=n>time</span><span class=o>.</span><span class=n>time</span><span class=p>()</span></span>
<span class=code-line> <span class=n>time</span><span class=o>.</span><span class=n>sleep</span><span class=p>(</span><span class=n>delay</span><span class=p>)</span></span>
</pre></div> <p>и небольшой доработке&nbsp;callback-метода:</p> <div class=highlight><pre><span class=code-line><span></span><span class=c1># Конструктор, при запуске процесс запустит</span></span>
<span class=code-line><span class=c1># функцию с указанными параметрами</span></span>
<span class=code-line><span class=n>work_process</span> <span class=o>=</span> <span class=n>multiprocessing</span><span class=o>.</span><span class=n>Process</span><span class=p>(</span></span>
<span class=code-line> <span class=n>target</span><span class=o>=</span><span class=n>real_work</span><span class=p>,</span></span>
<span class=code-line> <span class=n>args</span><span class=o>=</span><span class=p>(</span><span class=n>body</span><span class=p>,</span> <span class=p>)</span></span>
<span class=code-line><span class=p>)</span></span>
<span class=code-line><span class=n>work_process</span><span class=o>.</span><span class=n>start</span><span class=p>()</span> <span class=c1># процесс нужно явно запустить</span></span>
<span class=code-line><span class=k>while</span> <span class=bp>True</span><span class=p>:</span></span>
<span class=code-line> <span class=n>ch</span><span class=o>.</span><span class=n>connection</span><span class=o>.</span><span class=n>sleep</span><span class=p>(</span><span class=mi>1</span><span class=p>)</span></span>
<span class=code-line> <span class=c1># пока поток работает, используем функцию pika</span></span>
<span class=code-line> <span class=c1># для обработки необходимых сообщений</span></span>
<span class=code-line> <span class=k>if</span> <span class=ow>not</span> <span class=n>work_process</span><span class=o>.</span><span class=n>is_alive</span><span class=p>():</span></span>
<span class=code-line> <span class=k>break</span></span>
</pre></div> <p>Тогда готовый код будет выглядеть&nbsp;так:</p> <div class=highlight><pre><span class=code-line><span></span><span class=ch>#!/usr/bin/python</span></span>
<span class=code-line><span class=c1># -*- coding: utf-8 -*-</span></span>
<span class=code-line></span>
<span class=code-line><span class=n>__author__</span> <span class=o>=</span> <span class=s2>&quot;Aleksey Lobanov&quot;</span></span>
<span class=code-line><span class=n>__license__</span> <span class=o>=</span> <span class=s2>&quot;MIT&quot;</span></span>
<span class=code-line></span>
<span class=code-line><span class=kn>import</span> <span class=nn>sys</span></span>
<span class=code-line><span class=kn>import</span> <span class=nn>time</span></span>
<span class=code-line><span class=kn>import</span> <span class=nn>math</span></span>
<span class=code-line><span class=kn>import</span> <span class=nn>multiprocessing</span></span>
<span class=code-line></span>
<span class=code-line><span class=kn>import</span> <span class=nn>pika</span></span>
<span class=code-line></span>
<span class=code-line></span>
<span class=code-line><span class=k>def</span> <span class=nf>real_work</span><span class=p>(</span><span class=n>body</span><span class=p>):</span></span>
<span class=code-line> <span class=n>delay</span> <span class=o>=</span> <span class=nb>int</span><span class=p>(</span><span class=n>body</span><span class=p>)</span></span>
<span class=code-line> <span class=n>begin_at</span> <span class=o>=</span> <span class=n>time</span><span class=o>.</span><span class=n>time</span><span class=p>()</span></span>
<span class=code-line> <span class=n>time</span><span class=o>.</span><span class=n>sleep</span><span class=p>(</span><span class=n>delay</span><span class=p>)</span></span>
<span class=code-line></span>
<span class=code-line></span>
<span class=code-line><span class=k>def</span> <span class=nf>callback</span><span class=p>(</span><span class=n>ch</span><span class=p>,</span> <span class=n>method</span><span class=p>,</span> <span class=n>properties</span><span class=p>,</span> <span class=n>body</span><span class=p>):</span></span>
<span class=code-line> <span class=k>print</span><span class=p>(</span><span class=s2>&quot; [x] Received </span><span class=si>%r</span><span class=s2>&quot;</span> <span class=o>%</span> <span class=n>body</span><span class=p>)</span></span>
<span class=code-line> <span class=n>work_process</span> <span class=o>=</span> <span class=n>multiprocessing</span><span class=o>.</span><span class=n>Process</span><span class=p>(</span></span>
<span class=code-line> <span class=n>target</span><span class=o>=</span><span class=n>real_work</span><span class=p>,</span></span>
<span class=code-line> <span class=n>args</span><span class=o>=</span><span class=p>(</span><span class=n>body</span><span class=p>,</span> <span class=p>)</span></span>
<span class=code-line> <span class=p>)</span></span>
<span class=code-line> <span class=n>work_process</span><span class=o>.</span><span class=n>start</span><span class=p>()</span></span>
<span class=code-line> <span class=k>while</span> <span class=bp>True</span><span class=p>:</span></span>
<span class=code-line> <span class=n>ch</span><span class=o>.</span><span class=n>connection</span><span class=o>.</span><span class=n>sleep</span><span class=p>(</span><span class=mi>1</span><span class=p>)</span></span>
<span class=code-line> <span class=k>if</span> <span class=ow>not</span> <span class=n>work_process</span><span class=o>.</span><span class=n>is_alive</span><span class=p>():</span></span>
<span class=code-line> <span class=k>break</span></span>
<span class=code-line></span>
<span class=code-line> <span class=k>print</span><span class=p>(</span><span class=s2>&quot; [x] Finished &quot;</span> <span class=o>+</span> <span class=n>body</span><span class=p>)</span></span>
<span class=code-line></span>
<span class=code-line></span>
<span class=code-line><span class=k>if</span> <span class=vm>__name__</span> <span class=o>==</span> <span class=s2>&quot;__main__&quot;</span><span class=p>:</span></span>
<span class=code-line> <span class=n>connection</span> <span class=o>=</span> <span class=n>pika</span><span class=o>.</span><span class=n>BlockingConnection</span><span class=p>(</span><span class=n>pika</span><span class=o>.</span><span class=n>ConnectionParameters</span><span class=p>(</span><span class=n>host</span><span class=o>=</span><span class=s1>&#39;localhost&#39;</span><span class=p>))</span></span>
<span class=code-line> <span class=n>channel</span> <span class=o>=</span> <span class=n>connection</span><span class=o>.</span><span class=n>channel</span><span class=p>()</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>queue_declare</span><span class=p>(</span><span class=n>queue</span><span class=o>=</span><span class=s1>&#39;demo.hello&#39;</span><span class=p>)</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>basic_consume</span><span class=p>(</span></span>
<span class=code-line> <span class=n>callback</span><span class=p>,</span></span>
<span class=code-line> <span class=n>queue</span><span class=o>=</span><span class=s1>&#39;demo.hello&#39;</span><span class=p>,</span></span>
<span class=code-line> <span class=n>no_ack</span><span class=o>=</span><span class=bp>True</span></span>
<span class=code-line> <span class=p>)</span></span>
<span class=code-line></span>
<span class=code-line> <span class=k>print</span><span class=p>(</span><span class=s1>&#39; [*] Waiting for messages. To exit press CTRL+C&#39;</span><span class=p>)</span></span>
<span class=code-line> <span class=n>channel</span><span class=o>.</span><span class=n>start_consuming</span><span class=p>()</span></span>
</pre></div> </section> <hr> <aside class=post-meta> <p>Категория: <a href=../../category/python.html>Python</a></p> <p>Теги: <a href=../../tag/python.html>Python</a>, <a href=../../tag/rabbitmq.html>RabbitMQ</a>, </p> </aside> <hr> </article> </li> </ol> </div> </div> <script type=text/javascript>
var _paq = _paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
@@ -143,4 +143,4 @@
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script> <script src="../../theme/js/main.js"></script></body></html>
</script> <script src=../../theme/js/main.js></script> </body> </html>

View File

@@ -1,8 +1,8 @@
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class="no-js"> <head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><title>Экспорт партий с Lichess</title><meta name="description" content><meta name="viewport" content="width=device-width"><link rel="stylesheet" href="../../theme/css/normalize.css"><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel="stylesheet"><link rel="stylesheet" href="../../theme/css/font-awesome.min.css"><link rel="stylesheet" href="../../theme/css/main.css"><link rel="stylesheet" href="../../theme/css/blog.css"><link rel="stylesheet" href="../../theme/css/github.css"><link href="https://likemath.ru/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Блог 529 Atom Feed"><link href="https://likemath.ru/feeds/all.rss.xml" type="application/rss+xml" rel="alternate" title="Блог 529 RSS Feed"><script src="../../theme/js/vendor/modernizr-2.6.2.min.js"></script></head><body><!--[if lt IE 7]>
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class=no-js> <!--<![endif]--> <head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><title>Экспорт партий с Lichess</title><meta name=description content><meta name=viewport content="width=device-width"><link rel=stylesheet href=../../theme/css/normalize.css><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel=stylesheet><link rel=stylesheet href=../../theme/css/font-awesome.min.css><link rel=stylesheet href=../../theme/css/main.css><link rel=stylesheet href=../../theme/css/blog.css><link rel=stylesheet href=../../theme/css/github.css><link href=https://likemath.ru/feeds/all.atom.xml type=application/atom+xml rel=alternate title="Блог 529 Atom Feed"><link href=https://likemath.ru/feeds/all.rss.xml type=application/rss+xml rel=alternate title="Блог 529 RSS Feed"><script src=../../theme/js/vendor/modernizr-2.6.2.min.js></script></head> <body> <!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]--><div id="wrapper"><header id="sidebar" class="side-shadow"><hgroup id="site-header"><a id="site-title" href="../.."><h2><i class="icon-pencil"></i> Блог 529</h2></a><p id="site-desc"> Project Euler и остальное </p></hgroup><nav><ul id="nav-links"><li><a href="../../">Главная</a></li><li><a href="../../pages/projects.html">Мои проекты</a></li><li><a href="../../pages/about.html">Об авторе</a></li><li><a href="../../feeds/feed.atom.xml">Atom feed</a></li></ul></nav><footer id="site-info"><p> Powered by Pelican. </p></footer></header><div id="post-container"><ol id="post-list"><li><article class="post-entry"><header class="entry-header"><time class="post-time" datetime="2017-07-17T13:35:00+03:00" pubdate> Пн 17 Июль 2017 </time><a href="../../posts/eksport-partii-s-lichess/" rel="bookmark"><h1>Экспорт партий с&nbsp;Lichess</h1></a></header><section class="post-content"><p>Одну партию с Lichess скачать довольно просто, это можно сделать на странице с самой игрой. Скачать все игры тоже несложно, для этого есть специальный <a href="https://database.lichess.org/">раздел</a>.</p><p>Если же хочется скачать свои партии, то я нашёл лишь одно <a href="https://github.com/cyanfish/lichess-scid">решение</a>. Последний коммит был сделан 2 года назад, также оно не учитывает ограничения <a href="https://github.com/ornicar/lila#http-api"><span class="caps">API</span></a>, и больше ~2000 партий, за раз скачать не&nbsp;получится.</p><p>Моей целью было написание альтернативы, которая бы работала быстрее и работала&nbsp;корректно.</p><p>Проблема формата решена довольно просто: <span class="caps">API</span> отдаёт <span class="caps">PGN</span>, значит его поддержка уже есть. Но это плохой формат для хранения и обработки. Поэтому нужно было выбрать что-то&nbsp;ещё.</p><p>Существует, практически, только Scid, если вы хотите хранить свои шахматные партии, используя открытый софт. В его дистрибутиве уже есть необходимые утилиты для конвертации pgn в свой&nbsp;формат.</p><p>Репозиторий <a href="https://github.com/AlekseyLobanov/lichess-export/">здесь</a>, а <a href="https://github.com/AlekseyLobanov/lichess-export/archive/master.zip">здесь</a> можно скачать файлы&nbsp;сразу.</p><h3>Возможности</h3><ol><li>Многопоточность</li><li>Сохранение как в <span class="caps">PGN</span>, так и в формат&nbsp;Scid</li><li>Для работы необходим только&nbsp;Python</li><li>Работает при любом числе&nbsp;партий</li></ol><h3>Установка</h3><div class="highlight"><pre><span class="code-line"><span></span>pip install grequests</span>
</pre></div><h3>Запуск</h3><div class="highlight"><pre><span class="code-line"><span></span>python lichess.py -n hippo23 -t pgn -o hippo23.pgn --threads <span class="m">6</span> </span>
</pre></div><p>Здесь <code>hippo23</code> мой ник, а <code>pgn</code> формат сохранения (доступен также <code>scid</code>). </p></section><hr><aside class="post-meta"><p>Категория: <a href="../../category/proekty.html">Проекты</a></p><p>Теги: <a href="../../tag/proekt.html">проект</a>, <a href="../../tag/shakhmaty.html">шахматы</a>, <a href="../../tag/lichess.html">lichess</a>, </p></aside><hr></article></li></ol></div></div><script type="text/javascript">
<![endif]--> <div id=wrapper> <header id=sidebar class=side-shadow> <hgroup id=site-header> <a id=site-title href=../..><h2><i class=icon-pencil></i> Блог 529</h2></a> <p id=site-desc> Project Euler и остальное </p> </hgroup> <nav> <ul id=nav-links> <li><a href=../../ >Главная</a></li> <li><a href=../../pages/projects.html>Мои проекты</a></li> <li><a href=../../pages/about.html>Об авторе</a></li> <li><a href=../../feeds/feed.atom.xml>Atom feed</a></li> </ul> </nav> <footer id=site-info> <p> Powered by Pelican. </p> </footer></header> <div id=post-container> <ol id=post-list> <li> <article class=post-entry> <header class=entry-header> <time class=post-time datetime=2017-07-17T13:35:00+03:00 pubdate> Пн 17 июля 2017 </time> <a href=../../posts/eksport-partii-s-lichess/ rel=bookmark><h1>Экспорт партий с&nbsp;Lichess</h1></a> </header> <section class=post-content> <p>Одну партию с Lichess скачать довольно просто, это можно сделать на странице с самой игрой. Скачать все игры тоже несложно, для этого есть специальный <a href=https://database.lichess.org/ >раздел</a>.</p> <p>Если же хочется скачать свои партии, то я нашёл лишь одно <a href=https://github.com/cyanfish/lichess-scid>решение</a>. Последний коммит был сделан 2 года назад, также оно не учитывает ограничения <a href=https://github.com/ornicar/lila#http-api><span class=caps>API</span></a>, и больше ~2000 партий, за раз скачать не&nbsp;получится.</p> <p>Моей целью было написание альтернативы, которая бы работала быстрее и работала&nbsp;корректно.</p> <p>Проблема формата решена довольно просто: <span class=caps>API</span> отдаёт <span class=caps>PGN</span>, значит его поддержка уже есть. Но это плохой формат для хранения и обработки. Поэтому нужно было выбрать что-то&nbsp;ещё.</p> <p>Существует, практически, только Scid, если вы хотите хранить свои шахматные партии, используя открытый софт. В его дистрибутиве уже есть необходимые утилиты для конвертации pgn в свой&nbsp;формат.</p> <p>Репозиторий <a href=https://github.com/AlekseyLobanov/lichess-export/ >здесь</a>, а <a href=https://github.com/AlekseyLobanov/lichess-export/archive/master.zip>здесь</a> можно скачать файлы&nbsp;сразу.</p> <h3>Возможности</h3> <ol> <li>Многопоточность</li> <li>Сохранение как в <span class=caps>PGN</span>, так и в формат&nbsp;Scid</li> <li>Для работы необходим только&nbsp;Python</li> <li>Работает при любом числе&nbsp;партий</li> </ol> <h3>Установка</h3> <div class=highlight><pre><span class=code-line><span></span>pip install grequests</span>
</pre></div> <h3>Запуск</h3> <div class=highlight><pre><span class=code-line><span></span>python lichess.py -n hippo23 -t pgn -o hippo23.pgn --threads <span class=m>6</span> </span>
</pre></div> <p>Здесь <code>hippo23</code> мой ник, а <code>pgn</code> формат сохранения (доступен также <code>scid</code>). </p> </section> <hr> <aside class=post-meta> <p>Категория: <a href=../../category/proekty.html>Проекты</a></p> <p>Теги: <a href=../../tag/proekt.html>проект</a>, <a href=../../tag/shakhmaty.html>шахматы</a>, <a href=../../tag/lichess.html>lichess</a>, </p> </aside> <hr> </article> </li> </ol> </div> </div> <script type=text/javascript>
var _paq = _paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
@@ -13,4 +13,4 @@
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script> <script src="../../theme/js/main.js"></script></body></html>
</script> <script src=../../theme/js/main.js></script> </body> </html>

View File

@@ -1,6 +1,6 @@
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class="no-js"> <head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><title>Ещё одно вычисление выражений</title><meta name="description" content><meta name="viewport" content="width=device-width"><link rel="stylesheet" href="../../theme/css/normalize.css"><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel="stylesheet"><link rel="stylesheet" href="../../theme/css/font-awesome.min.css"><link rel="stylesheet" href="../../theme/css/main.css"><link rel="stylesheet" href="../../theme/css/blog.css"><link rel="stylesheet" href="../../theme/css/github.css"><link href="https://likemath.ru/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Блог 529 Atom Feed"><link href="https://likemath.ru/feeds/all.rss.xml" type="application/rss+xml" rel="alternate" title="Блог 529 RSS Feed"><script src="../../theme/js/vendor/modernizr-2.6.2.min.js"></script></head><body><!--[if lt IE 7]>
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class=no-js> <!--<![endif]--> <head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><title>Ещё одно вычисление выражений</title><meta name=description content><meta name=viewport content="width=device-width"><link rel=stylesheet href=../../theme/css/normalize.css><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel=stylesheet><link rel=stylesheet href=../../theme/css/font-awesome.min.css><link rel=stylesheet href=../../theme/css/main.css><link rel=stylesheet href=../../theme/css/blog.css><link rel=stylesheet href=../../theme/css/github.css><link href=https://likemath.ru/feeds/all.atom.xml type=application/atom+xml rel=alternate title="Блог 529 Atom Feed"><link href=https://likemath.ru/feeds/all.rss.xml type=application/rss+xml rel=alternate title="Блог 529 RSS Feed"><script src=../../theme/js/vendor/modernizr-2.6.2.min.js></script></head> <body> <!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]--><div id="wrapper"><header id="sidebar" class="side-shadow"><hgroup id="site-header"><a id="site-title" href="../.."><h2><i class="icon-pencil"></i> Блог 529</h2></a><p id="site-desc"> Project Euler и остальное </p></hgroup><nav><ul id="nav-links"><li><a href="../../">Главная</a></li><li><a href="../../pages/projects.html">Мои проекты</a></li><li><a href="../../pages/about.html">Об авторе</a></li><li><a href="../../feeds/feed.atom.xml">Atom feed</a></li></ul></nav><footer id="site-info"><p> Powered by Pelican. </p></footer></header><div id="post-container"><ol id="post-list"><li><article class="post-entry"><header class="entry-header"><time class="post-time" datetime="2015-07-03T17:40:00+03:00" pubdate> Пт 03 Июль 2015 </time><a href="../../posts/eshchio-odno-vychislenie-vyrazhenii/" rel="bookmark"><h1>Ещё одно вычисление&nbsp;выражений</h1></a></header><section class="post-content"><p>Задачка кажется не очень сложной, даже, если не знать как её делать (я не знал). Целью является быстрое вычисление чего-то типа <code>4 * ( 5 + 7 ^ 4)</code>. Для это я парсил исходную строку в список токенов, а затем непосредственно вычислял, что&nbsp;получится.</p><p>Я решил, что проще всего будет реализовать (а мне потом и понять) алгоритм, когда после каждого действия будет выполняться некий &#8220;хороший&#8221; инвариант. Первое что приходит в голову &#8212; это то, что истинность выражение после выполнения операции не меняется (<span class="caps">TITO</span> соблюдается). То есть выражение <code>3 + 5</code> можно заменить на <code>8</code> или хотя бы на <code>4 * 2</code>.</p><p>Непосредственно сама обработка является несколькими проходами, так что в каждом проходе мы избавляемся от операций одного приоритета. <code>4 + 5 * 3</code> заменяется на <code>4 + 15, 7 - 5 * 2^3</code> заменяется на <code>7 - 5*8</code>. Таким образом, каждый цикл тривиален, и легко задавать приоритеты&nbsp;операций.</p><p>Если использовать один список как контейнер для токенов и при работе изменять непосредственно его, сохраняя указанные инварианты, то сложность получается <span class="math">\(O\left( N \right)\)</span>, где <span class="math">\(N\)</span> &#8212; число&nbsp;токенов.</p><p>Времени на непосредственно кодирование ушло часа три-четыре, но в это время не входит продумывание мелких&nbsp;деталей.</p><p>Всё написано на C++11. Исходники лежат на <a href="https://github.com/AlekseyLobanov/ExprEvaler">GitHub</a> и <a href="https://bitbucket.org/Begemot23/exprevaler">BitBucket</a>.</p><script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
<![endif]--> <div id=wrapper> <header id=sidebar class=side-shadow> <hgroup id=site-header> <a id=site-title href=../..><h2><i class=icon-pencil></i> Блог 529</h2></a> <p id=site-desc> Project Euler и остальное </p> </hgroup> <nav> <ul id=nav-links> <li><a href=../../ >Главная</a></li> <li><a href=../../pages/projects.html>Мои проекты</a></li> <li><a href=../../pages/about.html>Об авторе</a></li> <li><a href=../../feeds/feed.atom.xml>Atom feed</a></li> </ul> </nav> <footer id=site-info> <p> Powered by Pelican. </p> </footer></header> <div id=post-container> <ol id=post-list> <li> <article class=post-entry> <header class=entry-header> <time class=post-time datetime=2015-07-03T17:40:00+03:00 pubdate> Пт 03 июля 2015 </time> <a href=../../posts/eshchio-odno-vychislenie-vyrazhenii/ rel=bookmark><h1>Ещё одно вычисление&nbsp;выражений</h1></a> </header> <section class=post-content> <p>Задачка кажется не очень сложной, даже, если не знать как её делать (я не знал). Целью является быстрое вычисление чего-то типа <code>4 * ( 5 + 7 ^ 4)</code>. Для это я парсил исходную строку в список токенов, а затем непосредственно вычислял, что&nbsp;получится.</p> <p>Я решил, что проще всего будет реализовать (а мне потом и понять) алгоритм, когда после каждого действия будет выполняться некий &#8220;хороший&#8221; инвариант. Первое что приходит в голову &#8212; это то, что истинность выражение после выполнения операции не меняется (<span class=caps>TITO</span> соблюдается). То есть выражение <code>3 + 5</code> можно заменить на <code>8</code> или хотя бы на <code>4 * 2</code>.</p> <p>Непосредственно сама обработка является несколькими проходами, так что в каждом проходе мы избавляемся от операций одного приоритета. <code>4 + 5 * 3</code> заменяется на <code>4 + 15, 7 - 5 * 2^3</code> заменяется на <code>7 - 5*8</code>. Таким образом, каждый цикл тривиален, и легко задавать приоритеты&nbsp;операций.</p> <p>Если использовать один список как контейнер для токенов и при работе изменять непосредственно его, сохраняя указанные инварианты, то сложность получается <span class=math>\(O\left( N \right)\)</span>, где <span class=math>\(N\)</span> &#8212; число&nbsp;токенов.</p> <p>Времени на непосредственно кодирование ушло часа три-четыре, но в это время не входит продумывание мелких&nbsp;деталей.</p> <p>Всё написано на C++11. Исходники лежат на <a href=https://github.com/AlekseyLobanov/ExprEvaler>GitHub</a> и <a href=https://bitbucket.org/Begemot23/exprevaler>BitBucket</a>.</p> <script type=text/javascript>if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
@@ -14,11 +14,14 @@
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
@@ -32,6 +35,8 @@
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
@@ -52,9 +57,11 @@
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script></section><hr><aside class="post-meta"><p>Категория: <a href="../../category/proekty.html">Проекты</a></p><p>Теги: <a href="../../tag/proekt.html">проект</a>, </p></aside><hr></article></li></ol></div></div><script type="text/javascript">
</script> </section> <hr> <aside class=post-meta> <p>Категория: <a href=../../category/proekty.html>Проекты</a></p> <p>Теги: <a href=../../tag/proekt.html>проект</a>, </p> </aside> <hr> </article> </li> </ol> </div> </div> <script type=text/javascript>
var _paq = _paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
@@ -65,4 +72,4 @@
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script> <script src="../../theme/js/main.js"></script></body></html>
</script> <script src=../../theme/js/main.js></script> </body> </html>

File diff suppressed because one or more lines are too long

Binary file not shown.

File diff suppressed because one or more lines are too long

View File

@@ -1,6 +1,6 @@
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class="no-js"> <head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><title>Мой первый пост или зачем этот блог?</title><meta name="description" content><meta name="viewport" content="width=device-width"><link rel="stylesheet" href="../../theme/css/normalize.css"><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel="stylesheet"><link rel="stylesheet" href="../../theme/css/font-awesome.min.css"><link rel="stylesheet" href="../../theme/css/main.css"><link rel="stylesheet" href="../../theme/css/blog.css"><link rel="stylesheet" href="../../theme/css/github.css"><link href="https://likemath.ru/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Блог 529 Atom Feed"><link href="https://likemath.ru/feeds/all.rss.xml" type="application/rss+xml" rel="alternate" title="Блог 529 RSS Feed"><script src="../../theme/js/vendor/modernizr-2.6.2.min.js"></script></head><body><!--[if lt IE 7]>
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class=no-js> <!--<![endif]--> <head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><title>Мой первый пост или зачем этот блог?</title><meta name=description content><meta name=viewport content="width=device-width"><link rel=stylesheet href=../../theme/css/normalize.css><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel=stylesheet><link rel=stylesheet href=../../theme/css/font-awesome.min.css><link rel=stylesheet href=../../theme/css/main.css><link rel=stylesheet href=../../theme/css/blog.css><link rel=stylesheet href=../../theme/css/github.css><link href=https://likemath.ru/feeds/all.atom.xml type=application/atom+xml rel=alternate title="Блог 529 Atom Feed"><link href=https://likemath.ru/feeds/all.rss.xml type=application/rss+xml rel=alternate title="Блог 529 RSS Feed"><script src=../../theme/js/vendor/modernizr-2.6.2.min.js></script></head> <body> <!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]--><div id="wrapper"><header id="sidebar" class="side-shadow"><hgroup id="site-header"><a id="site-title" href="../.."><h2><i class="icon-pencil"></i> Блог 529</h2></a><p id="site-desc"> Project Euler и остальное </p></hgroup><nav><ul id="nav-links"><li><a href="../../">Главная</a></li><li><a href="../../pages/projects.html">Мои проекты</a></li><li><a href="../../pages/about.html">Об авторе</a></li><li><a href="../../feeds/feed.atom.xml">Atom feed</a></li></ul></nav><footer id="site-info"><p> Powered by Pelican. </p></footer></header><div id="post-container"><ol id="post-list"><li><article class="post-entry"><header class="entry-header"><time class="post-time" datetime="2015-04-17T13:35:00+03:00" pubdate> Пт 17 Апрель 2015 </time><a href="../../posts/moi-pervyi-post-ili-zachem-etot-blog/" rel="bookmark"><h1>Мой первый пост или зачем этот&nbsp;блог?</h1></a></header><section class="post-content"><p>Есть несколько причин появления этого блога. Самая главная &#8212; хочется как-то использовать домен, который я купил изначально для красивой почты. Но она не единственная. Мне также хочется делиться частью того, что я делаю на платформе, которую я могу контролировать. Ну и контактные данные. Хорошо, когда все в одном&nbsp;месте.</p><p>Пока я планирую публиковать свои, возможно не лучшие, но рабочие решения для задач из Project Euler (<a href="https://projecteuler.net/">projecteuler.net</a>). На самом деле, решения значительного числа задач уже есть в Сети, но на английском. Хотя, это и не есть большая&nbsp;проблема.</p></section><hr><aside class="post-meta"><p>Категория: <a href="../../category/misc.html">misc</a></p><p>Теги: <a href="../../tag/blog.html">блог</a>, </p></aside><hr></article></li></ol></div></div><script type="text/javascript">
<![endif]--> <div id=wrapper> <header id=sidebar class=side-shadow> <hgroup id=site-header> <a id=site-title href=../..><h2><i class=icon-pencil></i> Блог 529</h2></a> <p id=site-desc> Project Euler и остальное </p> </hgroup> <nav> <ul id=nav-links> <li><a href=../../ >Главная</a></li> <li><a href=../../pages/projects.html>Мои проекты</a></li> <li><a href=../../pages/about.html>Об авторе</a></li> <li><a href=../../feeds/feed.atom.xml>Atom feed</a></li> </ul> </nav> <footer id=site-info> <p> Powered by Pelican. </p> </footer></header> <div id=post-container> <ol id=post-list> <li> <article class=post-entry> <header class=entry-header> <time class=post-time datetime=2015-04-17T13:35:00+03:00 pubdate> Пт 17 апреля 2015 </time> <a href=../../posts/moi-pervyi-post-ili-zachem-etot-blog/ rel=bookmark><h1>Мой первый пост или зачем этот&nbsp;блог?</h1></a> </header> <section class=post-content> <p>Есть несколько причин появления этого блога. Самая главная &#8212; хочется как-то использовать домен, который я купил изначально для красивой почты. Но она не единственная. Мне также хочется делиться частью того, что я делаю на платформе, которую я могу контролировать. Ну и контактные данные. Хорошо, когда все в одном&nbsp;месте.</p> <p>Пока я планирую публиковать свои, возможно не лучшие, но рабочие решения для задач из Project Euler (<a href=https://projecteuler.net/ >projecteuler.net</a>). На самом деле, решения значительного числа задач уже есть в Сети, но на английском. Хотя, это и не есть большая&nbsp;проблема.</p> </section> <hr> <aside class=post-meta> <p>Категория: <a href=../../category/misc.html>misc</a></p> <p>Теги: <a href=../../tag/blog.html>блог</a>, </p> </aside> <hr> </article> </li> </ol> </div> </div> <script type=text/javascript>
var _paq = _paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
@@ -11,4 +11,4 @@
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script> <script src="../../theme/js/main.js"></script></body></html>
</script> <script src=../../theme/js/main.js></script> </body> </html>

View File

@@ -1,29 +1,29 @@
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class="no-js"> <head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><title>Моё решение задачи 134</title><meta name="description" content><meta name="viewport" content="width=device-width"><link rel="stylesheet" href="../../theme/css/normalize.css"><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel="stylesheet"><link rel="stylesheet" href="../../theme/css/font-awesome.min.css"><link rel="stylesheet" href="../../theme/css/main.css"><link rel="stylesheet" href="../../theme/css/blog.css"><link rel="stylesheet" href="../../theme/css/github.css"><link href="https://likemath.ru/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Блог 529 Atom Feed"><link href="https://likemath.ru/feeds/all.rss.xml" type="application/rss+xml" rel="alternate" title="Блог 529 RSS Feed"><script src="../../theme/js/vendor/modernizr-2.6.2.min.js"></script></head><body><!--[if lt IE 7]>
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class=no-js> <!--<![endif]--> <head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><title>Моё решение задачи 134</title><meta name=description content><meta name=viewport content="width=device-width"><link rel=stylesheet href=../../theme/css/normalize.css><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel=stylesheet><link rel=stylesheet href=../../theme/css/font-awesome.min.css><link rel=stylesheet href=../../theme/css/main.css><link rel=stylesheet href=../../theme/css/blog.css><link rel=stylesheet href=../../theme/css/github.css><link href=https://likemath.ru/feeds/all.atom.xml type=application/atom+xml rel=alternate title="Блог 529 Atom Feed"><link href=https://likemath.ru/feeds/all.rss.xml type=application/rss+xml rel=alternate title="Блог 529 RSS Feed"><script src=../../theme/js/vendor/modernizr-2.6.2.min.js></script></head> <body> <!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]--><div id="wrapper"><header id="sidebar" class="side-shadow"><hgroup id="site-header"><a id="site-title" href="../.."><h2><i class="icon-pencil"></i> Блог 529</h2></a><p id="site-desc"> Project Euler и остальное </p></hgroup><nav><ul id="nav-links"><li><a href="../../">Главная</a></li><li><a href="../../pages/projects.html">Мои проекты</a></li><li><a href="../../pages/about.html">Об авторе</a></li><li><a href="../../feeds/feed.atom.xml">Atom feed</a></li></ul></nav><footer id="site-info"><p> Powered by Pelican. </p></footer></header><div id="post-container"><ol id="post-list"><li><article class="post-entry"><header class="entry-header"><time class="post-time" datetime="2016-10-30T17:40:00+03:00" pubdate> Вс 30 Октябрь 2016 </time><a href="../../posts/moio-reshenie-zadachi-134/" rel="bookmark"><h1>Моё решение задачи&nbsp;134</h1></a></header><section class="post-content"><p>Назовём <em>порождающим</em> для двух последовательных простых <span class="math">\(p_1 &lt; p_2\)</span> наименьшее натуральное число, что оно закачивается на <span class="math">\(p_1\)</span> и при этом делится на <span class="math">\(p_2\)</span>. Необходимо найти сумму порождающих для всех <span class="math">\(p_1 \in \left[ 5; 10^6&nbsp;\right]\)</span></p><p>Например, если <span class="math">\(p_1 = 19\)</span>, то следующее простое <span class="math">\(p_2 = 23\)</span>. Тогда порождающим будет число <span class="math">\(1219\)</span>, при этом <span class="math">\(1219 \: \vdots \: 23\)</span>.</p><p>Полное условие можно найти <a href="https://projecteuler.net/problem=134">тут</a></p><p>Несмотря на то, что сложность задачи 45%, для её решения достаточно выписать&nbsp;условие.</p><p>Пусть <span class="math">\(p_1\)</span> содержит в себе <span class="math">\(k\)</span> цифр, т.е. <span class="math">\(n = r \cdot 10^k + p_1\)</span>, где <span class="math">\(r\)</span> &#8212; какое-то натуральное число с отрезка <span class="math">\(\left[ 1; p_2-1&nbsp;\right]\)</span></p><p>Давайте посчитаем остатки по модулю <span class="math">\(p_2\)</span>: <span class="math">\(n \equiv r \cdot 10^k + p_1 \equiv 0\)</span>. Отсюда получим явную формулу для <span class="math">\(r\)</span>: </p><div class="math">$$ r \equiv -p_1 \cdot 10^{-k} \equiv -p_1 \cdot 10^{p_2 -1-k} $$</div><p>Комментарии:</p><ol><li>Так как <span class="math">\(a^p \equiv a \mod p\)</span>, то верно что <span class="math">\(a^{-k} \equiv a^{p -1-k} \mod&nbsp;p\)</span></li><li>Это всё бессмысленно, если не знать про <a href="https://ru.wikipedia.org/wiki/Алгоритмы_быстрогоозведения_в_степень">алгоритм быстрого возведения в степень</a>, который делает асимптотическую сложность возведения в степень&nbsp;логарифмической.</li></ol><p>У нас есть явная формула для порождающего, и мы знаем как её быстро посчитать. Ниже приведён код на Python с использованием <a href="http://www.sympy.org/ru/">sympy</a>.</p><div class="highlight"><pre><span class="code-line"><span></span><span class="kn">from</span> <span class="nn">sympy</span> <span class="kn">import</span> <span class="n">primerange</span> <span class="c1"># для получения простых чисел</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="c1"># быстрое возведение в степень по модулю</span></span>
<span class="code-line"><span class="k">def</span> <span class="nf">fast_pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">modulo</span><span class="p">):</span></span>
<span class="code-line"> <span class="k">if</span> <span class="n">y</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span></span>
<span class="code-line"> <span class="k">return</span> <span class="mi">1</span></span>
<span class="code-line"> <span class="n">p</span> <span class="o">=</span> <span class="n">fast_pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="o">//</span> <span class="mi">2</span><span class="p">,</span> <span class="n">modulo</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">p</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span> <span class="n">p</span><span class="p">)</span> <span class="o">%</span> <span class="n">modulo</span></span>
<span class="code-line"> <span class="k">if</span> <span class="n">y</span> <span class="o">%</span> <span class="mi">2</span><span class="p">:</span></span>
<span class="code-line"> <span class="n">p</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span> <span class="n">x</span><span class="p">)</span> <span class="o">%</span> <span class="n">modulo</span></span>
<span class="code-line"> <span class="k">return</span> <span class="n">p</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="c1"># нам нужно первое простое, которое больше 10^6 -- 10^6+3</span></span>
<span class="code-line"><span class="n">primes</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">primerange</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span><span class="mi">10</span><span class="o">**</span><span class="mi">6</span><span class="o">+</span><span class="mi">4</span><span class="p">))</span> </span>
<span class="code-line"></span>
<span class="code-line"><span class="n">sm</span> <span class="o">=</span> <span class="mi">0</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">primes</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">):</span></span>
<span class="code-line"> <span class="n">digs</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">primes</span><span class="p">[</span><span class="n">i</span><span class="p">]))</span> <span class="c1"># количество цифр</span></span>
<span class="code-line"> <span class="n">r</span> <span class="o">=</span> <span class="p">(</span><span class="n">primes</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span><span class="o">**</span><span class="mi">2</span> <span class="o">-</span> <span class="n">primes</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">*</span> <span class="n">fast_pow</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">primes</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">digs</span><span class="p">,</span> <span class="n">primes</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]))</span> <span class="o">%</span> <span class="n">primes</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span></span>
<span class="code-line"> <span class="n">sm</span> <span class="o">+=</span> <span class="n">r</span> <span class="o">*</span> <span class="mi">10</span><span class="o">**</span><span class="n">digs</span> <span class="o">+</span> <span class="n">primes</span><span class="p">[</span><span class="n">i</span><span class="p">]</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">print</span><span class="p">(</span><span class="s1">&#39;Result is {}&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">sm</span><span class="p">))</span></span>
</pre></div><p>Ответ: <strong>18613426663617118</strong></p><script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
<![endif]--> <div id=wrapper> <header id=sidebar class=side-shadow> <hgroup id=site-header> <a id=site-title href=../..><h2><i class=icon-pencil></i> Блог 529</h2></a> <p id=site-desc> Project Euler и остальное </p> </hgroup> <nav> <ul id=nav-links> <li><a href=../../ >Главная</a></li> <li><a href=../../pages/projects.html>Мои проекты</a></li> <li><a href=../../pages/about.html>Об авторе</a></li> <li><a href=../../feeds/feed.atom.xml>Atom feed</a></li> </ul> </nav> <footer id=site-info> <p> Powered by Pelican. </p> </footer></header> <div id=post-container> <ol id=post-list> <li> <article class=post-entry> <header class=entry-header> <time class=post-time datetime=2016-10-30T17:40:00+03:00 pubdate> Вс 30 октября 2016 </time> <a href=../../posts/moio-reshenie-zadachi-134/ rel=bookmark><h1>Моё решение задачи&nbsp;134</h1></a> </header> <section class=post-content> <p>Назовём <em>порождающим</em> для двух последовательных простых <span class=math>\(p_1 &lt; p_2\)</span> наименьшее натуральное число, что оно закачивается на <span class=math>\(p_1\)</span> и при этом делится на <span class=math>\(p_2\)</span>. Необходимо найти сумму порождающих для всех <span class=math>\(p_1 \in \left[ 5; 10^6&nbsp;\right]\)</span></p> <p>Например, если <span class=math>\(p_1 = 19\)</span>, то следующее простое <span class=math>\(p_2 = 23\)</span>. Тогда порождающим будет число <span class=math>\(1219\)</span>, при этом <span class=math>\(1219 \: \vdots \: 23\)</span>.</p> <p>Полное условие можно найти <a href="https://projecteuler.net/problem=134">тут</a></p> <p>Несмотря на то, что сложность задачи 45%, для её решения достаточно выписать&nbsp;условие.</p> <p>Пусть <span class=math>\(p_1\)</span> содержит в себе <span class=math>\(k\)</span> цифр, т.е. <span class=math>\(n = r \cdot 10^k + p_1\)</span>, где <span class=math>\(r\)</span> &#8212; какое-то натуральное число с отрезка <span class=math>\(\left[ 1; p_2-1&nbsp;\right]\)</span></p> <p>Давайте посчитаем остатки по модулю <span class=math>\(p_2\)</span>: <span class=math>\(n \equiv r \cdot 10^k + p_1 \equiv 0\)</span>. Отсюда получим явную формулу для <span class=math>\(r\)</span>: </p> <div class=math>$$ r \equiv -p_1 \cdot 10^{-k} \equiv -p_1 \cdot 10^{p_2 -1-k} $$</div> <p>Комментарии:</p> <ol> <li>Так как <span class=math>\(a^p \equiv a \mod p\)</span>, то верно что <span class=math>\(a^{-k} \equiv a^{p -1-k} \mod&nbsp;p\)</span></li> <li>Это всё бессмысленно, если не знать про <a href=https://ru.wikipedia.org/wiki/Алгоритмы_быстрогоозведения_в_степень>алгоритм быстрого возведения в степень</a>, который делает асимптотическую сложность возведения в степень&nbsp;логарифмической.</li> </ol> <p>У нас есть явная формула для порождающего, и мы знаем как её быстро посчитать. Ниже приведён код на Python с использованием <a href=http://www.sympy.org/ru/ >sympy</a>.</p> <div class=highlight><pre><span class=code-line><span></span><span class=kn>from</span> <span class=nn>sympy</span> <span class=kn>import</span> <span class=n>primerange</span> <span class=c1># для получения простых чисел</span></span>
<span class=code-line></span>
<span class=code-line><span class=c1># быстрое возведение в степень по модулю</span></span>
<span class=code-line><span class=k>def</span> <span class=nf>fast_pow</span><span class=p>(</span><span class=n>x</span><span class=p>,</span> <span class=n>y</span><span class=p>,</span> <span class=n>modulo</span><span class=p>):</span></span>
<span class=code-line> <span class=k>if</span> <span class=n>y</span> <span class=o>==</span> <span class=mi>0</span><span class=p>:</span></span>
<span class=code-line> <span class=k>return</span> <span class=mi>1</span></span>
<span class=code-line> <span class=n>p</span> <span class=o>=</span> <span class=n>fast_pow</span><span class=p>(</span><span class=n>x</span><span class=p>,</span> <span class=n>y</span> <span class=o>//</span> <span class=mi>2</span><span class=p>,</span> <span class=n>modulo</span><span class=p>)</span></span>
<span class=code-line> <span class=n>p</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>*</span> <span class=n>p</span><span class=p>)</span> <span class=o>%</span> <span class=n>modulo</span></span>
<span class=code-line> <span class=k>if</span> <span class=n>y</span> <span class=o>%</span> <span class=mi>2</span><span class=p>:</span></span>
<span class=code-line> <span class=n>p</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>*</span> <span class=n>x</span><span class=p>)</span> <span class=o>%</span> <span class=n>modulo</span></span>
<span class=code-line> <span class=k>return</span> <span class=n>p</span></span>
<span class=code-line></span>
<span class=code-line><span class=c1># нам нужно первое простое, которое больше 10^6 -- 10^6+3</span></span>
<span class=code-line><span class=n>primes</span> <span class=o>=</span> <span class=nb>list</span><span class=p>(</span><span class=n>primerange</span><span class=p>(</span><span class=mi>5</span><span class=p>,</span><span class=mi>10</span><span class=o>**</span><span class=mi>6</span><span class=o>+</span><span class=mi>4</span><span class=p>))</span> </span>
<span class=code-line></span>
<span class=code-line><span class=n>sm</span> <span class=o>=</span> <span class=mi>0</span></span>
<span class=code-line></span>
<span class=code-line><span class=k>for</span> <span class=n>i</span> <span class=ow>in</span> <span class=nb>range</span><span class=p>(</span><span class=nb>len</span><span class=p>(</span><span class=n>primes</span><span class=p>)</span> <span class=o>-</span> <span class=mi>1</span><span class=p>):</span></span>
<span class=code-line> <span class=n>digs</span> <span class=o>=</span> <span class=nb>len</span><span class=p>(</span><span class=nb>str</span><span class=p>(</span><span class=n>primes</span><span class=p>[</span><span class=n>i</span><span class=p>]))</span> <span class=c1># количество цифр</span></span>
<span class=code-line> <span class=n>r</span> <span class=o>=</span> <span class=p>(</span><span class=n>primes</span><span class=p>[</span><span class=n>i</span><span class=o>+</span><span class=mi>1</span><span class=p>]</span><span class=o>**</span><span class=mi>2</span> <span class=o>-</span> <span class=n>primes</span><span class=p>[</span><span class=n>i</span><span class=p>]</span> <span class=o>*</span> <span class=n>fast_pow</span><span class=p>(</span><span class=mi>10</span><span class=p>,</span> <span class=n>primes</span><span class=p>[</span><span class=n>i</span><span class=o>+</span><span class=mi>1</span><span class=p>]</span> <span class=o>-</span> <span class=mi>1</span> <span class=o>-</span> <span class=n>digs</span><span class=p>,</span> <span class=n>primes</span><span class=p>[</span><span class=n>i</span><span class=o>+</span><span class=mi>1</span><span class=p>]))</span> <span class=o>%</span> <span class=n>primes</span><span class=p>[</span><span class=n>i</span><span class=o>+</span><span class=mi>1</span><span class=p>]</span></span>
<span class=code-line> <span class=n>sm</span> <span class=o>+=</span> <span class=n>r</span> <span class=o>*</span> <span class=mi>10</span><span class=o>**</span><span class=n>digs</span> <span class=o>+</span> <span class=n>primes</span><span class=p>[</span><span class=n>i</span><span class=p>]</span></span>
<span class=code-line></span>
<span class=code-line><span class=k>print</span><span class=p>(</span><span class=s1>&#39;Result is {}&#39;</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>sm</span><span class=p>))</span></span>
</pre></div> <p>Ответ: <strong>18613426663617118</strong></p> <script type=text/javascript>if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
@@ -37,11 +37,14 @@
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
@@ -55,6 +58,8 @@
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
@@ -75,9 +80,11 @@
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script></section><hr><aside class="post-meta"><p>Категория: <a href="../../category/project-euler.html">Project Euler</a></p><p>Теги: <a href="../../tag/project-euler.html">Project Euler</a>, <a href="../../tag/python.html">Python</a>, <a href="../../tag/sympy.html">sympy</a>, </p></aside><hr></article></li></ol></div></div><script type="text/javascript">
</script> </section> <hr> <aside class=post-meta> <p>Категория: <a href=../../category/project-euler.html>Project Euler</a></p> <p>Теги: <a href=../../tag/project-euler.html>Project Euler</a>, <a href=../../tag/python.html>Python</a>, <a href=../../tag/sympy.html>sympy</a>, </p> </aside> <hr> </article> </li> </ol> </div> </div> <script type=text/javascript>
var _paq = _paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
@@ -88,4 +95,4 @@
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script> <script src="../../theme/js/main.js"></script></body></html>
</script> <script src=../../theme/js/main.js></script> </body> </html>

View File

@@ -1,112 +1,112 @@
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class="no-js"> <head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><title>Моё решение задачи 146</title><meta name="description" content><meta name="viewport" content="width=device-width"><link rel="stylesheet" href="../../theme/css/normalize.css"><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel="stylesheet"><link rel="stylesheet" href="../../theme/css/font-awesome.min.css"><link rel="stylesheet" href="../../theme/css/main.css"><link rel="stylesheet" href="../../theme/css/blog.css"><link rel="stylesheet" href="../../theme/css/github.css"><link href="https://likemath.ru/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Блог 529 Atom Feed"><link href="https://likemath.ru/feeds/all.rss.xml" type="application/rss+xml" rel="alternate" title="Блог 529 RSS Feed"><script src="../../theme/js/vendor/modernizr-2.6.2.min.js"></script></head><body><!--[if lt IE 7]>
<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--><!--[if gt IE 8]><!--><html class=no-js> <!--<![endif]--> <head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><title>Моё решение задачи 146</title><meta name=description content><meta name=viewport content="width=device-width"><link rel=stylesheet href=../../theme/css/normalize.css><link href="https://fonts.googleapis.com/css?family=Forum|Oswald|PT+Sans|Philosopher|Ubuntu+Mono" rel=stylesheet><link rel=stylesheet href=../../theme/css/font-awesome.min.css><link rel=stylesheet href=../../theme/css/main.css><link rel=stylesheet href=../../theme/css/blog.css><link rel=stylesheet href=../../theme/css/github.css><link href=https://likemath.ru/feeds/all.atom.xml type=application/atom+xml rel=alternate title="Блог 529 Atom Feed"><link href=https://likemath.ru/feeds/all.rss.xml type=application/rss+xml rel=alternate title="Блог 529 RSS Feed"><script src=../../theme/js/vendor/modernizr-2.6.2.min.js></script></head> <body> <!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]--><div id="wrapper"><header id="sidebar" class="side-shadow"><hgroup id="site-header"><a id="site-title" href="../.."><h2><i class="icon-pencil"></i> Блог 529</h2></a><p id="site-desc"> Project Euler и остальное </p></hgroup><nav><ul id="nav-links"><li><a href="../../">Главная</a></li><li><a href="../../pages/projects.html">Мои проекты</a></li><li><a href="../../pages/about.html">Об авторе</a></li><li><a href="../../feeds/feed.atom.xml">Atom feed</a></li></ul></nav><footer id="site-info"><p> Powered by Pelican. </p></footer></header><div id="post-container"><ol id="post-list"><li><article class="post-entry"><header class="entry-header"><time class="post-time" datetime="2016-10-21T17:40:00+03:00" pubdate> Пт 21 Октябрь 2016 </time><a href="../../posts/moio-reshenie-zadachi-146/" rel="bookmark"><h1>Моё решение задачи&nbsp;146</h1></a></header><section class="post-content"><p>Необходимо найти сумму всех натуральных <span class="math">\(n\)</span>, что <span class="math">\(n^2+1\)</span>, <span class="math">\(n^2+3\)</span>, <span class="math">\(n^2+7\)</span>, <span class="math">\(n^2+9\)</span>, <span class="math">\(n^2+13\)</span>, и <span class="math">\(n^2+27\)</span> будут <em>последовательными</em> простыми&nbsp;числами.</p><p>Полное условие можно найти <a href="https://projecteuler.net/problem=146">тут</a></p><p>Хочется отметить, что сложность у задачи 50%, а на текущий момент её решило меньше 4000 человек. Тем не менее, мне она показалось простой. Простейшее решение отработало очень&nbsp;быстро.</p><p>Для начала, можно отметить, что в лоб проверять условие очень долго. Проверять на простоту числа порядка <span class="math">\(10^{15}\)</span> достаточно сложно, поэтому их нужно как-то&nbsp;отсеять.</p><p>Самое простое &#8212; не рассматривать те <span class="math">\(n\)</span>, что хотя бы одно из <span class="math">\(n^2+1\)</span>, <span class="math">\(n^2+3\)</span>, <span class="math">\(n^2+7\)</span>, <span class="math">\(n^2+9\)</span>, <span class="math">\(n^2+13\)</span>, и <span class="math">\(n^2+27\)</span> будет заведомо делиться на какое-то маленькое простое число. Это даёт достаточно хорошие результаты: из 150 миллионов чисел, после отсеивания по простым числам <span class="math">\(&lt; 3000\)</span> (этот параметр я подбирал уже после решения задач: если он слишком маленький, то будет слишком много проверок на простоту, если же слишком большой, то мы делаем слишком много работы, чтобы отсеять несколько чисел), останется меньше <span class="math">\(2000\)</span> чисел. Их уже можно проверить&nbsp;непосредственно. </p><p>Тогда алгоритм может быть&nbsp;таким:</p><ol><li>Находим простые числа меньше <span class="math">\(3000\)</span>.</li><li>Для каждого из них находим допустимые&nbsp;остатки.</li><li>Для каждого из чисел от <span class="math">\(1\)</span> до <span class="math">\(n\)</span> проверяем, что остатки по всем простым&nbsp;хорошие.</li><li>Непосредственно проверяем условие. Важно не забыть проверить <strong>не</strong>простоту оставшихся нечётных чисел из диапазона <span class="math">\(n^2 + 1 \ldots n^2 + 27\)</span> там могут быть (и будут!) другие простые&nbsp;числа.</li></ol><p>Непосредственно сам поиск такой клики можно реализовать тривиально. Ниже мой код на C++11 с использованием библиотек Flint и primesieve. Распараллеливание хоть и просится, но смысла не имеет, т.к. я получил ответ менее, чем за 5&nbsp;секунд.</p><div class="highlight"><pre><span class="code-line"><span></span><span class="cm">/*</span></span>
<span class="code-line"><span class="cm"> * Problem 146 on Project Euler</span></span>
<span class="code-line"><span class="cm"> * Aleksey Lobanov (c) 2016</span></span>
<span class="code-line"><span class="cm"> */</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&lt;iostream&gt;</span><span class="cp"></span></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&lt;vector&gt;</span><span class="cp"></span></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&lt;cstdint&gt;</span><span class="cp"></span></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&lt;set&gt;</span><span class="cp"></span></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&lt;iomanip&gt;</span><span class="cp"></span></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&lt;algorithm&gt;</span><span class="cp"></span></span>
<span class="code-line"></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&quot;fmpzxx.h&quot;</span><span class="cp"></span></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&quot;arithxx.h&quot;</span><span class="cp"></span></span>
<span class="code-line"></span>
<span class="code-line"><span class="cp">#include</span> <span class="cpf">&quot;primesieve.hpp&quot;</span><span class="cp"></span></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">using</span> <span class="k">namespace</span> <span class="n">std</span><span class="p">;</span></span>
<span class="code-line"><span class="k">using</span> <span class="k">namespace</span> <span class="n">flint</span><span class="p">;</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kt">bool</span> <span class="nf">is_prime</span><span class="p">(</span><span class="kt">int64_t</span> <span class="n">num</span><span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="n">fmpz_factorxx</span> <span class="n">fact</span><span class="p">;</span></span>
<span class="code-line"> <span class="n">fact</span><span class="p">.</span><span class="n">set_factor</span><span class="p">(</span><span class="n">num</span><span class="p">);</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="n">fact</span><span class="p">.</span><span class="n">size</span><span class="p">()</span> <span class="o">!=</span> <span class="mi">1</span> <span class="p">)</span></span>
<span class="code-line"> <span class="k">return</span> <span class="nb">false</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="n">fact</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">)</span> </span>
<span class="code-line"> <span class="k">return</span> <span class="nb">false</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">return</span> <span class="nb">true</span><span class="p">;</span></span>
<span class="code-line"><span class="p">}</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kt">bool</span> <span class="nf">is_possible</span><span class="p">(</span><span class="kt">int64_t</span> <span class="n">num</span><span class="p">,</span> <span class="k">const</span> <span class="n">vector</span><span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span> <span class="o">&amp;</span><span class="n">to_add</span><span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="k">auto</span> <span class="o">&amp;&amp;</span><span class="nl">add</span><span class="p">:</span> <span class="n">to_add</span><span class="p">)</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="o">!</span><span class="n">is_prime</span><span class="p">(</span><span class="n">num</span><span class="o">*</span><span class="n">num</span> <span class="o">+</span> <span class="n">add</span><span class="p">)</span> <span class="p">)</span></span>
<span class="code-line"> <span class="k">return</span> <span class="nb">false</span><span class="p">;</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="c1">// primes must be consecutive</span></span>
<span class="code-line"> <span class="c1">// so we need check, that other numbers like n^2 + i is not primes</span></span>
<span class="code-line"> <span class="n">vector</span><span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span> <span class="n">other_adds</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="kt">size_t</span> <span class="n">i</span> <span class="o">=</span> <span class="n">to_add</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="mi">2</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">to_add</span><span class="p">[</span><span class="n">to_add</span><span class="p">.</span><span class="n">size</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span><span class="p">];</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">2</span><span class="p">)</span> </span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="o">!</span><span class="n">binary_search</span><span class="p">(</span><span class="n">to_add</span><span class="p">.</span><span class="n">begin</span><span class="p">(),</span> <span class="n">to_add</span><span class="p">.</span><span class="n">end</span><span class="p">(),</span> <span class="n">i</span><span class="p">)</span> <span class="p">)</span></span>
<span class="code-line"> <span class="n">other_adds</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">i</span><span class="p">);</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="k">auto</span> <span class="o">&amp;&amp;</span><span class="nl">add</span><span class="p">:</span> <span class="n">other_adds</span><span class="p">)</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="n">is_prime</span><span class="p">(</span><span class="n">num</span><span class="o">*</span><span class="n">num</span> <span class="o">+</span> <span class="n">add</span><span class="p">)</span> <span class="p">)</span></span>
<span class="code-line"> <span class="k">return</span> <span class="nb">false</span><span class="p">;</span> </span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">return</span> <span class="nb">true</span><span class="p">;</span></span>
<span class="code-line"><span class="p">}</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span></span>
<span class="code-line"> <span class="k">const</span> <span class="n">vector</span><span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span> <span class="n">to_add</span> <span class="o">=</span> <span class="p">{</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">13</span><span class="p">,</span> <span class="mi">27</span><span class="p">};</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">const</span> <span class="kt">int64_t</span> <span class="n">MAX_N</span> <span class="o">=</span> <span class="mi">1l</span><span class="o">*</span><span class="mi">150</span><span class="o">*</span><span class="mi">1000</span><span class="o">*</span><span class="mi">1000</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">const</span> <span class="kt">int64_t</span> <span class="n">MAX_PRIME</span> <span class="o">=</span> <span class="mi">3000</span><span class="p">;</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">vector</span><span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span> <span class="n">sieve_primes</span><span class="p">;</span></span>
<span class="code-line"> <span class="n">primesieve</span><span class="o">::</span><span class="n">generate_primes</span><span class="p">(</span><span class="n">MAX_PRIME</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">sieve_primes</span><span class="p">);</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">vector</span><span class="o">&lt;</span> <span class="n">vector</span> <span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span> <span class="o">&gt;</span> <span class="n">good_remainders</span><span class="p">;</span> </span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="k">auto</span> <span class="o">&amp;&amp;</span><span class="nl">prime</span><span class="p">:</span> <span class="n">sieve_primes</span><span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="n">set</span><span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span> <span class="n">remainders</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">for</span><span class="p">(</span><span class="kt">int64_t</span> <span class="n">rem</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">rem</span> <span class="o">&lt;</span> <span class="n">prime</span><span class="p">;</span> <span class="o">++</span><span class="n">rem</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">remainders</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">rem</span><span class="p">);</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">set</span><span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span> <span class="n">base_remainders</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="k">auto</span> <span class="o">&amp;&amp;</span><span class="nl">base</span><span class="p">:</span> <span class="n">to_add</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">base_remainders</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">base</span> <span class="o">%</span> <span class="n">prime</span><span class="p">);</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="kt">int64_t</span> <span class="n">rem</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">rem</span> <span class="o">&lt;</span> <span class="n">prime</span><span class="p">;</span> <span class="o">++</span><span class="n">rem</span><span class="p">)</span></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="k">auto</span> <span class="o">&amp;&amp;</span><span class="nl">base_rem</span><span class="p">:</span> <span class="n">base_remainders</span><span class="p">)</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="p">(</span><span class="n">rem</span><span class="o">*</span><span class="n">rem</span> <span class="o">+</span> <span class="n">base_rem</span><span class="p">)</span> <span class="o">%</span> <span class="n">prime</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">)</span></span>
<span class="code-line"> <span class="n">remainders</span><span class="p">.</span><span class="n">erase</span><span class="p">(</span><span class="n">rem</span><span class="p">);</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">good_remainders</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">vector</span><span class="o">&lt;</span><span class="kt">int64_t</span><span class="o">&gt;</span><span class="p">(</span><span class="n">remainders</span><span class="p">.</span><span class="n">begin</span><span class="p">(),</span> <span class="n">remainders</span><span class="p">.</span><span class="n">end</span><span class="p">()));</span></span>
<span class="code-line"> <span class="p">}</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="kt">size_t</span> <span class="n">cnt</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span></span>
<span class="code-line"> <span class="kt">size_t</span> <span class="n">sum</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="cm">/* WARNING</span></span>
<span class="code-line"><span class="cm"> * for small n can be that</span></span>
<span class="code-line"><span class="cm"> * n^2 + [1 or 3 or .. ] is prime in sieve primes</span></span>
<span class="code-line"><span class="cm"> * but there is only one n &lt; 315410 is 10,</span></span>
<span class="code-line"><span class="cm"> * so we need add 10 to n</span></span>
<span class="code-line"><span class="cm"> */</span></span>
<span class="code-line"> <span class="n">sum</span> <span class="o">+=</span> <span class="mi">10</span><span class="p">;</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="kt">int64_t</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">MAX_N</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="kt">bool</span> <span class="n">is_good</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">for</span> <span class="p">(</span><span class="kt">size_t</span> <span class="n">prime_ind</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">prime_ind</span> <span class="o">&lt;</span> <span class="n">sieve_primes</span><span class="p">.</span><span class="n">size</span><span class="p">();</span> <span class="o">++</span><span class="n">prime_ind</span><span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="o">!</span><span class="n">binary_search</span><span class="p">(</span><span class="n">good_remainders</span><span class="p">[</span><span class="n">prime_ind</span><span class="p">].</span><span class="n">begin</span><span class="p">(),</span> <span class="n">good_remainders</span><span class="p">[</span><span class="n">prime_ind</span><span class="p">].</span><span class="n">end</span><span class="p">(),</span> <span class="n">i</span> <span class="o">%</span> <span class="n">sieve_primes</span><span class="p">[</span><span class="n">prime_ind</span><span class="p">])</span> <span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="n">is_good</span> <span class="o">=</span> <span class="nb">false</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">break</span><span class="p">;</span></span>
<span class="code-line"> <span class="p">}</span></span>
<span class="code-line"> <span class="p">}</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="n">is_good</span> <span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="n">cnt</span><span class="o">++</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">if</span> <span class="p">(</span> <span class="n">is_possible</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">to_add</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="n">sum</span> <span class="o">+=</span> <span class="n">i</span><span class="p">;</span></span>
<span class="code-line"> <span class="p">}</span> </span>
<span class="code-line"> <span class="p">}</span></span>
<span class="code-line"> <span class="p">}</span></span>
<span class="code-line"> <span class="n">cout</span> <span class="o">&lt;&lt;</span> <span class="s">&quot;count = &quot;</span> <span class="o">&lt;&lt;</span> <span class="n">cnt</span> <span class="o">&lt;&lt;</span> <span class="n">endl</span><span class="p">;</span></span>
<span class="code-line"> <span class="n">cout</span> <span class="o">&lt;&lt;</span> <span class="s">&quot;Result is: &quot;</span> <span class="o">&lt;&lt;</span> <span class="n">sum</span> <span class="o">&lt;&lt;</span> <span class="n">endl</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span></span>
<span class="code-line"><span class="p">}</span></span>
</pre></div><p>Ответ: <strong>676333270</strong></p><script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
<![endif]--> <div id=wrapper> <header id=sidebar class=side-shadow> <hgroup id=site-header> <a id=site-title href=../..><h2><i class=icon-pencil></i> Блог 529</h2></a> <p id=site-desc> Project Euler и остальное </p> </hgroup> <nav> <ul id=nav-links> <li><a href=../../ >Главная</a></li> <li><a href=../../pages/projects.html>Мои проекты</a></li> <li><a href=../../pages/about.html>Об авторе</a></li> <li><a href=../../feeds/feed.atom.xml>Atom feed</a></li> </ul> </nav> <footer id=site-info> <p> Powered by Pelican. </p> </footer></header> <div id=post-container> <ol id=post-list> <li> <article class=post-entry> <header class=entry-header> <time class=post-time datetime=2016-10-21T17:40:00+03:00 pubdate> Пт 21 октября 2016 </time> <a href=../../posts/moio-reshenie-zadachi-146/ rel=bookmark><h1>Моё решение задачи&nbsp;146</h1></a> </header> <section class=post-content> <p>Необходимо найти сумму всех натуральных <span class=math>\(n\)</span>, что <span class=math>\(n^2+1\)</span>, <span class=math>\(n^2+3\)</span>, <span class=math>\(n^2+7\)</span>, <span class=math>\(n^2+9\)</span>, <span class=math>\(n^2+13\)</span>, и <span class=math>\(n^2+27\)</span> будут <em>последовательными</em> простыми&nbsp;числами.</p> <p>Полное условие можно найти <a href="https://projecteuler.net/problem=146">тут</a></p> <p>Хочется отметить, что сложность у задачи 50%, а на текущий момент её решило меньше 4000 человек. Тем не менее, мне она показалось простой. Простейшее решение отработало очень&nbsp;быстро.</p> <p>Для начала, можно отметить, что в лоб проверять условие очень долго. Проверять на простоту числа порядка <span class=math>\(10^{15}\)</span> достаточно сложно, поэтому их нужно как-то&nbsp;отсеять.</p> <p>Самое простое &#8212; не рассматривать те <span class=math>\(n\)</span>, что хотя бы одно из <span class=math>\(n^2+1\)</span>, <span class=math>\(n^2+3\)</span>, <span class=math>\(n^2+7\)</span>, <span class=math>\(n^2+9\)</span>, <span class=math>\(n^2+13\)</span>, и <span class=math>\(n^2+27\)</span> будет заведомо делиться на какое-то маленькое простое число. Это даёт достаточно хорошие результаты: из 150 миллионов чисел, после отсеивания по простым числам <span class=math>\(&lt; 3000\)</span> (этот параметр я подбирал уже после решения задач: если он слишком маленький, то будет слишком много проверок на простоту, если же слишком большой, то мы делаем слишком много работы, чтобы отсеять несколько чисел), останется меньше <span class=math>\(2000\)</span> чисел. Их уже можно проверить&nbsp;непосредственно. </p> <p>Тогда алгоритм может быть&nbsp;таким:</p> <ol> <li>Находим простые числа меньше <span class=math>\(3000\)</span>.</li> <li>Для каждого из них находим допустимые&nbsp;остатки.</li> <li>Для каждого из чисел от <span class=math>\(1\)</span> до <span class=math>\(n\)</span> проверяем, что остатки по всем простым&nbsp;хорошие.</li> <li>Непосредственно проверяем условие. Важно не забыть проверить <strong>не</strong>простоту оставшихся нечётных чисел из диапазона <span class=math>\(n^2 + 1 \ldots n^2 + 27\)</span> там могут быть (и будут!) другие простые&nbsp;числа.</li> </ol> <p>Непосредственно сам поиск такой клики можно реализовать тривиально. Ниже мой код на C++11 с использованием библиотек Flint и primesieve. Распараллеливание хоть и просится, но смысла не имеет, т.к. я получил ответ менее, чем за 5&nbsp;секунд.</p> <div class=highlight><pre><span class=code-line><span></span><span class=cm>/*</span></span>
<span class=code-line><span class=cm> * Problem 146 on Project Euler</span></span>
<span class=code-line><span class=cm> * Aleksey Lobanov (c) 2016</span></span>
<span class=code-line><span class=cm> */</span></span>
<span class=code-line></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&lt;iostream&gt;</span><span class=cp></span></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&lt;vector&gt;</span><span class=cp></span></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&lt;cstdint&gt;</span><span class=cp></span></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&lt;set&gt;</span><span class=cp></span></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&lt;iomanip&gt;</span><span class=cp></span></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&lt;algorithm&gt;</span><span class=cp></span></span>
<span class=code-line></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&quot;fmpzxx.h&quot;</span><span class=cp></span></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&quot;arithxx.h&quot;</span><span class=cp></span></span>
<span class=code-line></span>
<span class=code-line><span class=cp>#include</span> <span class=cpf>&quot;primesieve.hpp&quot;</span><span class=cp></span></span>
<span class=code-line></span>
<span class=code-line><span class=k>using</span> <span class=k>namespace</span> <span class=n>std</span><span class=p>;</span></span>
<span class=code-line><span class=k>using</span> <span class=k>namespace</span> <span class=n>flint</span><span class=p>;</span></span>
<span class=code-line></span>
<span class=code-line><span class=kt>bool</span> <span class=nf>is_prime</span><span class=p>(</span><span class=kt>int64_t</span> <span class=n>num</span><span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=n>fmpz_factorxx</span> <span class=n>fact</span><span class=p>;</span></span>
<span class=code-line> <span class=n>fact</span><span class=p>.</span><span class=n>set_factor</span><span class=p>(</span><span class=n>num</span><span class=p>);</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=n>fact</span><span class=p>.</span><span class=n>size</span><span class=p>()</span> <span class=o>!=</span> <span class=mi>1</span> <span class=p>)</span></span>
<span class=code-line> <span class=k>return</span> <span class=nb>false</span><span class=p>;</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=n>fact</span><span class=p>.</span><span class=n>exp</span><span class=p>(</span><span class=mi>0</span><span class=p>)</span> <span class=o>!=</span> <span class=mi>1</span><span class=p>)</span> </span>
<span class=code-line> <span class=k>return</span> <span class=nb>false</span><span class=p>;</span></span>
<span class=code-line> <span class=k>return</span> <span class=nb>true</span><span class=p>;</span></span>
<span class=code-line><span class=p>}</span></span>
<span class=code-line></span>
<span class=code-line><span class=kt>bool</span> <span class=nf>is_possible</span><span class=p>(</span><span class=kt>int64_t</span> <span class=n>num</span><span class=p>,</span> <span class=k>const</span> <span class=n>vector</span><span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span> <span class=o>&amp;</span><span class=n>to_add</span><span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=k>auto</span> <span class=o>&amp;&amp;</span><span class=nl>add</span><span class=p>:</span> <span class=n>to_add</span><span class=p>)</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=o>!</span><span class=n>is_prime</span><span class=p>(</span><span class=n>num</span><span class=o>*</span><span class=n>num</span> <span class=o>+</span> <span class=n>add</span><span class=p>)</span> <span class=p>)</span></span>
<span class=code-line> <span class=k>return</span> <span class=nb>false</span><span class=p>;</span></span>
<span class=code-line></span>
<span class=code-line> <span class=c1>// primes must be consecutive</span></span>
<span class=code-line> <span class=c1>// so we need check, that other numbers like n^2 + i is not primes</span></span>
<span class=code-line> <span class=n>vector</span><span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span> <span class=n>other_adds</span><span class=p>;</span></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=kt>size_t</span> <span class=n>i</span> <span class=o>=</span> <span class=n>to_add</span><span class=p>[</span><span class=mi>0</span><span class=p>]</span> <span class=o>+</span> <span class=mi>2</span><span class=p>;</span> <span class=n>i</span> <span class=o>&lt;</span> <span class=n>to_add</span><span class=p>[</span><span class=n>to_add</span><span class=p>.</span><span class=n>size</span><span class=p>()</span> <span class=o>-</span> <span class=mi>1</span><span class=p>];</span> <span class=n>i</span> <span class=o>+=</span> <span class=mi>2</span><span class=p>)</span> </span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=o>!</span><span class=n>binary_search</span><span class=p>(</span><span class=n>to_add</span><span class=p>.</span><span class=n>begin</span><span class=p>(),</span> <span class=n>to_add</span><span class=p>.</span><span class=n>end</span><span class=p>(),</span> <span class=n>i</span><span class=p>)</span> <span class=p>)</span></span>
<span class=code-line> <span class=n>other_adds</span><span class=p>.</span><span class=n>push_back</span><span class=p>(</span><span class=n>i</span><span class=p>);</span></span>
<span class=code-line></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=k>auto</span> <span class=o>&amp;&amp;</span><span class=nl>add</span><span class=p>:</span> <span class=n>other_adds</span><span class=p>)</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=n>is_prime</span><span class=p>(</span><span class=n>num</span><span class=o>*</span><span class=n>num</span> <span class=o>+</span> <span class=n>add</span><span class=p>)</span> <span class=p>)</span></span>
<span class=code-line> <span class=k>return</span> <span class=nb>false</span><span class=p>;</span> </span>
<span class=code-line></span>
<span class=code-line> <span class=k>return</span> <span class=nb>true</span><span class=p>;</span></span>
<span class=code-line><span class=p>}</span></span>
<span class=code-line></span>
<span class=code-line><span class=kt>int</span> <span class=nf>main</span><span class=p>()</span> <span class=p>{</span></span>
<span class=code-line> <span class=k>const</span> <span class=n>vector</span><span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span> <span class=n>to_add</span> <span class=o>=</span> <span class=p>{</span><span class=mi>1</span><span class=p>,</span> <span class=mi>3</span><span class=p>,</span> <span class=mi>7</span><span class=p>,</span> <span class=mi>9</span><span class=p>,</span> <span class=mi>13</span><span class=p>,</span> <span class=mi>27</span><span class=p>};</span></span>
<span class=code-line></span>
<span class=code-line> <span class=k>const</span> <span class=kt>int64_t</span> <span class=n>MAX_N</span> <span class=o>=</span> <span class=mi>1l</span><span class=o>*</span><span class=mi>150</span><span class=o>*</span><span class=mi>1000</span><span class=o>*</span><span class=mi>1000</span><span class=p>;</span></span>
<span class=code-line> <span class=k>const</span> <span class=kt>int64_t</span> <span class=n>MAX_PRIME</span> <span class=o>=</span> <span class=mi>3000</span><span class=p>;</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>vector</span><span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span> <span class=n>sieve_primes</span><span class=p>;</span></span>
<span class=code-line> <span class=n>primesieve</span><span class=o>::</span><span class=n>generate_primes</span><span class=p>(</span><span class=n>MAX_PRIME</span><span class=p>,</span> <span class=o>&amp;</span><span class=n>sieve_primes</span><span class=p>);</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>vector</span><span class=o>&lt;</span> <span class=n>vector</span> <span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span> <span class=o>&gt;</span> <span class=n>good_remainders</span><span class=p>;</span> </span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=k>auto</span> <span class=o>&amp;&amp;</span><span class=nl>prime</span><span class=p>:</span> <span class=n>sieve_primes</span><span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=n>set</span><span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span> <span class=n>remainders</span><span class=p>;</span></span>
<span class=code-line> <span class=k>for</span><span class=p>(</span><span class=kt>int64_t</span> <span class=n>rem</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span> <span class=n>rem</span> <span class=o>&lt;</span> <span class=n>prime</span><span class=p>;</span> <span class=o>++</span><span class=n>rem</span><span class=p>)</span></span>
<span class=code-line> <span class=n>remainders</span><span class=p>.</span><span class=n>insert</span><span class=p>(</span><span class=n>rem</span><span class=p>);</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>set</span><span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span> <span class=n>base_remainders</span><span class=p>;</span></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=k>auto</span> <span class=o>&amp;&amp;</span><span class=nl>base</span><span class=p>:</span> <span class=n>to_add</span><span class=p>)</span></span>
<span class=code-line> <span class=n>base_remainders</span><span class=p>.</span><span class=n>insert</span><span class=p>(</span><span class=n>base</span> <span class=o>%</span> <span class=n>prime</span><span class=p>);</span></span>
<span class=code-line></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=kt>int64_t</span> <span class=n>rem</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span> <span class=n>rem</span> <span class=o>&lt;</span> <span class=n>prime</span><span class=p>;</span> <span class=o>++</span><span class=n>rem</span><span class=p>)</span></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=k>auto</span> <span class=o>&amp;&amp;</span><span class=nl>base_rem</span><span class=p>:</span> <span class=n>base_remainders</span><span class=p>)</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=p>(</span><span class=n>rem</span><span class=o>*</span><span class=n>rem</span> <span class=o>+</span> <span class=n>base_rem</span><span class=p>)</span> <span class=o>%</span> <span class=n>prime</span> <span class=o>==</span> <span class=mi>0</span> <span class=p>)</span></span>
<span class=code-line> <span class=n>remainders</span><span class=p>.</span><span class=n>erase</span><span class=p>(</span><span class=n>rem</span><span class=p>);</span></span>
<span class=code-line></span>
<span class=code-line> <span class=n>good_remainders</span><span class=p>.</span><span class=n>push_back</span><span class=p>(</span><span class=n>vector</span><span class=o>&lt;</span><span class=kt>int64_t</span><span class=o>&gt;</span><span class=p>(</span><span class=n>remainders</span><span class=p>.</span><span class=n>begin</span><span class=p>(),</span> <span class=n>remainders</span><span class=p>.</span><span class=n>end</span><span class=p>()));</span></span>
<span class=code-line> <span class=p>}</span></span>
<span class=code-line></span>
<span class=code-line> <span class=kt>size_t</span> <span class=n>cnt</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span></span>
<span class=code-line> <span class=kt>size_t</span> <span class=n>sum</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span></span>
<span class=code-line></span>
<span class=code-line> <span class=cm>/* WARNING</span></span>
<span class=code-line><span class=cm> * for small n can be that</span></span>
<span class=code-line><span class=cm> * n^2 + [1 or 3 or .. ] is prime in sieve primes</span></span>
<span class=code-line><span class=cm> * but there is only one n &lt; 315410 is 10,</span></span>
<span class=code-line><span class=cm> * so we need add 10 to n</span></span>
<span class=code-line><span class=cm> */</span></span>
<span class=code-line> <span class=n>sum</span> <span class=o>+=</span> <span class=mi>10</span><span class=p>;</span></span>
<span class=code-line></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=kt>int64_t</span> <span class=n>i</span> <span class=o>=</span> <span class=mi>1</span><span class=p>;</span> <span class=n>i</span> <span class=o>&lt;</span> <span class=n>MAX_N</span><span class=p>;</span> <span class=o>++</span><span class=n>i</span><span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=kt>bool</span> <span class=n>is_good</span> <span class=o>=</span> <span class=nb>true</span><span class=p>;</span></span>
<span class=code-line> <span class=k>for</span> <span class=p>(</span><span class=kt>size_t</span> <span class=n>prime_ind</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span> <span class=n>prime_ind</span> <span class=o>&lt;</span> <span class=n>sieve_primes</span><span class=p>.</span><span class=n>size</span><span class=p>();</span> <span class=o>++</span><span class=n>prime_ind</span><span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=o>!</span><span class=n>binary_search</span><span class=p>(</span><span class=n>good_remainders</span><span class=p>[</span><span class=n>prime_ind</span><span class=p>].</span><span class=n>begin</span><span class=p>(),</span> <span class=n>good_remainders</span><span class=p>[</span><span class=n>prime_ind</span><span class=p>].</span><span class=n>end</span><span class=p>(),</span> <span class=n>i</span> <span class=o>%</span> <span class=n>sieve_primes</span><span class=p>[</span><span class=n>prime_ind</span><span class=p>])</span> <span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=n>is_good</span> <span class=o>=</span> <span class=nb>false</span><span class=p>;</span></span>
<span class=code-line> <span class=k>break</span><span class=p>;</span></span>
<span class=code-line> <span class=p>}</span></span>
<span class=code-line> <span class=p>}</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=n>is_good</span> <span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=n>cnt</span><span class=o>++</span><span class=p>;</span></span>
<span class=code-line> <span class=k>if</span> <span class=p>(</span> <span class=n>is_possible</span><span class=p>(</span><span class=n>i</span><span class=p>,</span> <span class=n>to_add</span><span class=p>)</span> <span class=p>)</span> <span class=p>{</span></span>
<span class=code-line> <span class=n>sum</span> <span class=o>+=</span> <span class=n>i</span><span class=p>;</span></span>
<span class=code-line> <span class=p>}</span> </span>
<span class=code-line> <span class=p>}</span></span>
<span class=code-line> <span class=p>}</span></span>
<span class=code-line> <span class=n>cout</span> <span class=o>&lt;&lt;</span> <span class=s>&quot;count = &quot;</span> <span class=o>&lt;&lt;</span> <span class=n>cnt</span> <span class=o>&lt;&lt;</span> <span class=n>endl</span><span class=p>;</span></span>
<span class=code-line> <span class=n>cout</span> <span class=o>&lt;&lt;</span> <span class=s>&quot;Result is: &quot;</span> <span class=o>&lt;&lt;</span> <span class=n>sum</span> <span class=o>&lt;&lt;</span> <span class=n>endl</span><span class=p>;</span></span>
<span class=code-line> <span class=k>return</span> <span class=mi>0</span><span class=p>;</span></span>
<span class=code-line><span class=p>}</span></span>
</pre></div> <p>Ответ: <strong>676333270</strong></p> <script type=text/javascript>if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
@@ -120,11 +120,14 @@
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
@@ -138,6 +141,8 @@
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
@@ -158,9 +163,11 @@
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script></section><hr><aside class="post-meta"><p>Категория: <a href="../../category/project-euler.html">Project Euler</a></p><p>Теги: <a href="../../tag/project-euler.html">Project Euler</a>, <a href="../../tag/c.html">c++</a>, <a href="../../tag/flint.html">FLINT</a>, </p></aside><hr></article></li></ol></div></div><script type="text/javascript">
</script> </section> <hr> <aside class=post-meta> <p>Категория: <a href=../../category/project-euler.html>Project Euler</a></p> <p>Теги: <a href=../../tag/project-euler.html>Project Euler</a>, <a href=../../tag/c.html>c++</a>, <a href=../../tag/flint.html>FLINT</a>, </p> </aside> <hr> </article> </li> </ol> </div> </div> <script type=text/javascript>
var _paq = _paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
@@ -171,4 +178,4 @@
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script> <script src="../../theme/js/main.js"></script></body></html>
</script> <script src=../../theme/js/main.js></script> </body> </html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long