tag:blogger.com,1999:blog-15079409127825440272024-03-17T02:30:40.015+06:00Délimiteur de donnéesEsprit curieuxDelimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.comBlogger201125tag:blogger.com,1999:blog-1507940912782544027.post-63030555926028411472021-07-03T22:19:00.001+06:002021-07-12T22:44:46.398+06:00Python 3.9 run Windows Server 2008 problem<pre><div><div><span style="font-family: Times New Roman;"><span style="white-space: normal;">According to https://www.python.org/downloads/windows/:</span></span></div><div><span style="font-family: Times New Roman;"><span style="white-space: normal;">Python 3.9 cannot be used on Windows 7 or earlier.</span></span></div><div style="font-family: "Times New Roman"; white-space: normal;">But we had a server with Windows Server 2008 R2 installed 🤷 and while running a project with Python 3.9 on Windows Server 2008 R2, we encountered the following issue: </div><div style="font-family: "Times New Roman"; white-space: normal;">python.exe run.py exited with -1073740777. It is Unknown Software Exception: 0xc0000417.</div></div><div><span style="font-family: "Times New Roman"; white-space: normal;">Failed process printed the next output:</span></div><div><span style="font-family: Times New Roman;"><span style="white-space: normal;"><br /></span></span></div>D:\Jenkins\workspace\TEST>"D:\Jenkins\workspace\TEST\project\python.exe" run.py
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'D:\Jenkins\workspace\TEST\project\python.exe'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '\xc5V\u3241Z'
sys.base_prefix = '.'
sys.base_exec_prefix = '.'
sys.platlibdir = 'lib'
sys.executable = '\xc5V\u3241Z'
sys.prefix = '.'
sys.exec_prefix = '.'
sys.path = [
'D:\\Jenkins\\workspace\\TEST\\project\\python39.zip',
'.\\DLLs',
'.\\lib',
'',
]
</pre>
<p>The variables <b>sys._base_executable</b> and <b>sys.executable</b> are corrupted and filled with arbitrary data.</p><p>We had <b>sitecustomize.py </b>file, with <b>sys.executable</b> variable update to required python.exe, but the problems still reproduced.</p><p>The only solution helped:</p><p>It is required to set the next environment variables (in our case we set them in Jenkins):</p><p>set PYTHONPATH=%PYTHONPATH%;%WORKSPACE%\project</p><p>set PYTHONHOME=%PYTHONHOME%;%WORKSPACE%\project</p><p>set __PYVENV_LAUNCHER__=%WORKSPACE%\project\python.exe</p>Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-48458387144957714942020-03-28T05:23:00.000+06:002020-04-05T05:54:51.738+06:00Blog's 10th birthday<div dir="ltr" style="text-align: left;" trbidi="on">
Happy 10-th birthday my dear blog!!! <br />
Time flies. <span class="tlid-translation translation" lang="en">I remember the first entry as if it were so recently</span>: <a href="https://delimitry.blogspot.com/2010/03/first-entry.html">https://delimitry.blogspot.com/2010/03/first-entry.html</a><br />
It were really interesting 10 years. <br />
Thanks to all the subscribers and readers!</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com2tag:blogger.com,1999:blog-1507940912782544027.post-76412742295478404102020-02-26T03:57:00.000+06:002020-03-17T03:57:37.558+06:00Underground<div dir="ltr" style="text-align: left;" trbidi="on">
Finished the book <a href="http://www.underground-book.net/">Underground</a> by Suelette Dreyfus. <br />Really exciting and interesting stories.<br />
Some situations are sound very funny today - with fast internet and huge disks.<br />
<span class="cr-widget-DesktopGlobalReviews" data-hook="cr-widget-DesktopGlobalReviews">Recommend it for anyone interested in hacking/freaking stories.<br />PS. Also check out a documentary <a href="https://en.wikipedia.org/wiki/In_the_Realm_of_the_Hackers">film</a> i</span>nspired by this book.<br />
<div class="text-wrap tlid-copy-target">
<div class="result-shield-container tlid-copy-target" tabindex="0">
<span class="tlid-translation-gender-indicator translation-gender-indicator"></span><span class="tlid-trans-verified-button trans-verified-button" role="button"></span></div>
</div>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-81838504135164823652019-12-20T03:51:00.002+06:002019-12-20T03:54:08.295+06:00JIT compilation for CPython<div dir="ltr" style="text-align: left;" trbidi="on">
My presentation from SPbPython meetup about simple self-made Just-in-time (JIT) compiler for CPython.<br />
<br />
<script async="" class="speakerdeck-embed" data-id="42514b3cb72944818b7cadd6eec1e257" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<br />
Link to slideshare:
<a href="https://www.slideshare.net/delimitry/jit-compilation-for-cpython">https://www.slideshare.net/delimitry/jit-compilation-for-cpython</a></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-35100276178736355702019-03-28T16:18:00.002+06:002019-03-28T16:19:07.127+06:00Gödel, Escher, Bach: An Eternal Golden Braid<div dir="ltr" style="text-align: left;" trbidi="on">
Yesterday I finished reading this great book by Douglas Richard Hofstadter.<br />
Great explanation of Gödel's incompleteness theorems with references to Bach's fugues and Escher's artworks, zen buddhism, molecular biology, NDA, RNA, programming and AI, reductionism and holism.<br />
Great food for thought! I like self-references and recursions! This book is worth re-reading to find something new.<br />
A lot of good references and further reading.<br />
<div>
Recommend to everyone interested in mathematics, art, music and AI.</div>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com1tag:blogger.com,1999:blog-1507940912782544027.post-73411893514394094522019-02-26T03:07:00.005+06:002019-02-26T03:07:57.618+06:00SNMP server in pure Python<div dir="ltr" style="text-align: left;" trbidi="on">
<span class="tlid-translation translation">Meet my new project: simple SNMP server in pure Python.<br />It can be used as a simple SNMP echo server or configured to respond to the requests with more complex logic.<br /><a href="https://github.com/delimitry/snmp-server">https://github.com/delimitry/snmp-server</a><br />This SNMP server is really useful if it is required to test SNMP client and mock values for OIDs</span><span class="tlid-translation translation">.</span></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com1tag:blogger.com,1999:blog-1507940912782544027.post-18051220642789211972018-11-16T16:06:00.000+06:002018-11-16T16:06:16.542+06:00Data Storage Systems<div dir="ltr" style="text-align: left;" trbidi="on">
My presentation from from PiterPy Meetup #10 Hardcore about the data structures used in databases for storing and retrieving data.<br />
Two approaches to data processing are considered: OLTP and OLAP.<br />
SQL, NoSQL and New SQL databases are discussed.<br />
The tradeoffs that the developers face when creating storage systems are shown.<br />
Also the methods of data storage and interaction with the database provides CPython are considered.<br />
The presentation and the list of references and books helps more easily navigate the data storage engines and understand which tool is better suited for a particular task.<br />
<br />
<br />
<script async="" class="speakerdeck-embed" data-id="01ef6971fe4f4a9799f3ba06b71288b7" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<br />
<br /></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-88243167402524919392018-10-16T14:25:00.000+06:002018-10-16T14:25:44.245+06:00SPARROW theorem and RUM Conjecture<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="color: #14171a; font-family: inherit;"><span style="background-color: white; font-size: 14px; white-space: pre-wrap;">I've found a post in RocksDB blog about <a href="http://rocksdb.blogspot.com/2013/10/SparrowTheorem.html">the SPAce, Read Or Write (SPARROW) theorem</a>, which states that:</span></span><br />
<span style="background-color: white; color: #14171a; font-family: inherit; font-size: 14px; white-space: pre-wrap;">1. Read Amplification (RA) is inversely related to Write Amplification (WA)</span><br />
<span style="color: #14171a; font-family: inherit;"><span style="background-color: white; font-size: 14px; white-space: pre-wrap;">2. Write Amplification (WA) is inversely related to Space Amplification (SA)</span></span><br />
<span style="font-family: inherit;"><br /></span>
<span style="color: #14171a; font-family: inherit;"><span style="background-color: white; font-size: 14px; white-space: pre-wrap;">Seems like the same, but more detailed principles are described in <a href="http://openproceedings.org/2016/conf/edbt/paper-12.pdf">RUM Conjecture</a> paper.</span></span></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-80356930093184314682018-10-14T18:22:00.000+06:002018-11-01T15:43:59.280+06:00Периодическая таблица структур данных<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Группа исследователей из Гарварда под руководством профессора Стратоса Идреоса опубликовала интересную работу "The Periodic Table of Data Structures" [1] (Периодическая таблица структур данных или даже Периодическая система структур данных), в которой они пытаются систематизировать существующие структуры данных путём их декомпозиции на части – основные принципы, то есть минимальные структуры данных (например связный список).<br />
Полученная в итоге модель позволяет увидеть какие структуры данных ещё остаются малоизученными или требуют оптимизации, а также получить лучшие результаты производительности путём комбинирования и настройки существующих структур данных.<br />
<br />
По мнению авторов, периодическая таблица структур данных, поможет ускорить исследования и принятие решений при проектировании и разработке новых структур данных.<br />
<br />
В работе также предлагается понятие design continuums (континуумы дизайна или континуумы проектирования) – часть
пространства проектирования структур данных, которое можно рассматривать
как единое, потому что оно определяется одним и тем же
набором основных принципов. Структурируя <span class="">пространство проектирования, можно заметить такие континуумы в классах структур
данных, которые традиционно считались принципиально разными.</span><br />
<br />
Данная работа тесно связана с другой работой под названием "Data Calculator" [2, 3] (Калькулятор данных), в которой исследователи пытаются разработать интерактивный, полуавтоматический подход для генерации и проектирования структур данных.<br />
<br />
Ранее они же предложили <a href="https://delimitry.blogspot.com/2018/03/rum.html">гипотезу RUM</a> про которую я уже писал в блоге.<br />
<div>
<br /></div>
<b>Ссылки:</b><br />
[1] S. Idreos, et al., “<a href="http://sites.computer.org/debull/A18sept/p64.pdf">The Periodic Table of Data Structures</a>,” Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, vol. 41, no. 3, pp. 64-75, 2018.<br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
[2] S. Idreos, K. Zoumpatianos, B. Hentschel, M. Kester, and D. Guo. The Data Calculator: Data Structure Design and Cost Synthesis from First Principles and Learned Cost Models. SIGMOD, 2018.<br />
[3] S. Idreos, K. Zoumpatianos, B. Hentschel, M. Kester, and D. Guo. <a href="https://arxiv.org/pdf/1808.02066.pdf">The Internals of the Data Calculator</a>, 2018.</div>
</div>
</div>
</div>
</div>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com1tag:blogger.com,1999:blog-1507940912782544027.post-89064312266908738052018-10-09T03:58:00.000+06:002018-10-09T03:58:25.952+06:00Epoch protection<div dir="ltr" style="text-align: left;" trbidi="on">
Epoch protection is a technique to avoid expensive synchronization between threads.<br /><br /><b>References:</b><br />[1] <a href="https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf">Keir Fraser: Practical lock-freedom</a><br />
[2] <a href="https://www.microsoft.com/en-us/research/uploads/prod/2018/03/faster-sigmod18.pdf">Faster: A Concurrent Key-Value Store with In-Place Updates</a><br />
<br />
<br /></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-63659874631822235432018-09-17T02:48:00.000+06:002018-10-09T03:17:28.200+06:00Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://pbs.twimg.com/media/Do_NzfqX4AE-Awv?format=jpg&name=small" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="680" data-original-width="517" height="400" src="https://pbs.twimg.com/media/Do_NzfqX4AE-Awv?format=jpg&name=small" width="303" /></a></div>
Finally I've finished reading this awesome book by Martin Kleppmann. It is one of the best technical books I have read. A Lot of useful information with great explanations and schemes.<br />
Special thanks for references after each chapter and of course beautiful maps between chapters :)<br />
This book sparked my interest in databases even more. Thank you Martin for such a good reading! <span style="text-align: center;">Can't stop to recommend it to everyone!</span></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com1tag:blogger.com,1999:blog-1507940912782544027.post-20517537359616181382018-09-13T03:24:00.002+06:002018-09-17T03:55:29.055+06:00LSM tree, Memtable and SSTable<div dir="ltr" style="text-align: left;" trbidi="on">
<b>LSM tree</b> (Log-structured merge-tree) is a data structure with performance characteristics that make it attractive for providing indexed access to files with high insert volume, such as transactional log data. LSM trees, like other search trees, maintain key-value pairs. LSM trees maintain data in two or more separate structures, each of which is optimized for its respective underlying storage medium; data is synchronized between the two structures efficiently, in batches [1].<br />
LSM trees are used in data stores such as Bigtable, HBase, LevelDB, RocksDB, ScyllaDB, Apache Cassandra, MongoDB, SQLite4, Tarantool, WiredTiger, InfluxDB, etc.<br />
<b><br /></b>
<b>Memtable</b> is in-memory data-structure that holds the data before it flushed to the SSTables.<br />
In LevelDB, RocksDB and Cassandra the implementation of Memtable is <span class="st">based on</span> skiplist [2] data-structure [3, 4, 5].<br />
<br />
<b>SSTable</b> (Sorted Strings Table) is a key-value based persistent, ordered immutable storage [6].<br />
The SSTable contains a sequence of blocks (typically 64 KB in size). At the end of the SSTable an index block is stored.<br />
<br />
References:<br />
[1] <a href="https://en.wikipedia.org/wiki/Log-structured_merge-tree">Log-structured merge-tree</a><br />
[2] <a href="https://en.wikipedia.org/wiki/Skip_list">Skip list</a><br />
[3] <a href="https://ayende.com/blog/161441/reviewing-leveldb-part-v-into-the-memtables-we-go">Reviewing LevelDB: Part V, into the MemTables we go</a><br />
[4] <a href="https://github.com/facebook/rocksdb/wiki/MemTable">MemTable</a><br />
[5] <a href="https://github.com/apache/cassandra/blob/trunk/doc/SASI.md">Apache Cassandra Github: SASIIndex</a><br />
[6] <a href="http://research.google.com/archive/bigtable.html">Bigtable: A Distributed Storage System for Structured Data</a></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-32105654333420858782018-08-20T17:27:00.002+06:002018-08-20T17:30:44.753+06:00PEPFS - Writing file system in CPython<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Writing file system in CPython<br />
<br />
On last PiterPy meetup I've made a talk about writing simple file system PEPFS in CPython using FUSE (File System In Userspace).<br />
<br />
The PEPFS project is available on github: <a href="https://github.com/delimitry/pepfs">https://github.com/delimitry/pepfs</a><br />
PEPFS is a simple read-only file system where files are CPython PEPs.<br />
To build file system a <code>fusepy</code> module was used.<br />
<div>
<br /></div>
Slides are available here <a href="https://speakerdeck.com/delimitry/writing-file-system-in-cpython">https://speakerdeck.com/delimitry/writing-file-system-in-cpython</a> and here <a href="https://www.slideshare.net/delimitry/writing-file-system-in-cpython">https://www.slideshare.net/delimitry/writing-file-system-in-cpython</a><br />
</div>
<script async="" class="speakerdeck-embed" data-id="0f6f5a60c9e645a49b2ca8d27e81a42f" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script><br />
<br /></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-87491886699021147622018-07-11T03:45:00.001+06:002018-07-11T03:52:57.927+06:00CPython logo<div dir="ltr" style="text-align: left;" trbidi="on">
On last Python meetup I've made a talk about CPython logo, its history, authors and meaning.<br />
<br />Slides are available here <a href="https://speakerdeck.com/delimitry/cpython-logo">https://speakerdeck.com/delimitry/cpython-logo</a> and here <a href="https://slideshare.net/delimitry/cpython-logo">https://slideshare.net/delimitry/cpython-logo</a><br />
<div>
<br />Please let me know if I you find any mistakes and inaccuracies<br /><br />
And here the embedded presentation:</div>
<script async="" class="speakerdeck-embed" data-id="74442743a57a4063b15fae9ec9769f71" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-57520229514568226042018-05-10T23:22:00.000+06:002018-05-14T16:11:14.426+06:00Seven Concurrency Models in Seven Weeks<div dir="ltr" style="text-align: left;" trbidi="on">
Just finished reading <a href="https://www.amazon.com/Seven-Concurrency-Models-Weeks-Programmers/dp/1937785653">Seven Concurrency Models in Seven Weeks</a> book by Paul Butcher.<br /><br />This book is the best overview of concurrent/parallel programming models I've ever seen.<br /><br />Highly recommend for everyone interested in distributed systems and concurrent/parallel programming paradigms.</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-36357597729115984202018-05-01T03:43:00.001+06:002018-09-18T03:28:49.899+06:00Python FLAG_REF and TYPE_REF<div dir="ltr" style="text-align: left;" trbidi="on">
Last week my friend asked me about 0x80<span class="pl-s"><span class="pl-cce"><b> </b>bit that sometimes set in Python marshalled objects data in *.pyc file.</span></span><br />
So I decided to understand what kind of flag it is and why it is used.<br />
<br />
This is <b>FLAG_REF</b> — a special flag that was added in Python 3.4, and used in conjunction with <b>TYPE_REF</b> type code value. For source code look at: <a href="https://github.com/python/cpython/blob/3.4/Python/marshal.c">https://github.com/python/cpython/blob/3.4/Python/marshal.c</a>.<br />
<div>
In the current version (Python 3.7) everything is the same.</div>
<div>
<br /></div>
<b><span class="pl-en">TYPE_REF</span> = <span class="pl-s"><span class="pl-pds">'</span>r<span class="pl-pds">'</span></span></b><br />
<div>
<b><span class="pl-en">FLAG_REF</span> = 0</b><span class="pl-s"><span class="pl-cce"><b>x80</b></span></span></div>
<br />
<span class="pl-s"><span class="pl-cce">All currently used type code values are < 0x80 (the highest type code is </span></span><span class="pl-en">TYPE_DICT =</span> <span class="pl-s"><span class="pl-pds">'</span>{<span class="pl-pds">', i.e. 0x7b)</span></span><span class="pl-s"><span class="pl-cce">, so it was decided to store this flag right in the highest bit of </span></span>type code value.<br />
<br />
Now let's see how this flag works.<br />
<br />
<span class="pl-en">When <b>marshal_load</b> reads *.pyc file with </span><b>read_object </b>function, it calls <b>r_object </b>to read marshalled binary data object by object.<br />
To understand the type of an object to read, a single byte with <b>code</b> is <a href="https://github.com/python/cpython/blob/150033d1599f55c10860a733d370707a3f7c444e/Python/marshal.c#L939">read</a>.<br />
<br />
<table class="highlight tab-size js-file-line-container" data-tab-size="8"><tbody>
<tr><td class="blob-code blob-code-inner js-file-line" id="LC866"><b> flag = code & FLAG_REF </b>to understand if <b>FLAG_REF</b> is set.</td>
</tr>
<tr>
</tr>
</tbody></table>
<b> type = code & ~FLAG_REF</b> to get only 7 bits (as all type code values are < 0x80).<br />
<br />
Also a special macro <b>R_REF</b> is <a href="https://github.com/python/cpython/blob/150033d1599f55c10860a733d370707a3f7c444e/Python/marshal.c#L960">defined</a>. This macro reads unmarshalled object into special <b>refs</b> list with <b>r_ref </b>function only if <b>FLAG_REF</b> is set.<br />
NB: on write (for mashalling) the <a href="https://github.com/python/cpython/blob/150033d1599f55c10860a733d370707a3f7c444e/Python/marshal.c#L82">hashtable</a>, instead of the list, is used.<br />
<br />
So when <b>type </b>is equal to <b>TYPE_REF</b>, then <a href="https://github.com/python/cpython/blob/150033d1599f55c10860a733d370707a3f7c444e/Python/marshal.c#L1446">only the reference index is read</a>, thus eliminating the need to unmarshall the already unmarshalled object again.</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-1185775072682270062018-04-24T19:20:00.001+06:002018-04-24T19:20:48.613+06:00Compressed Rich Text Format (RTF) aka LZFu compression and decompression in Python<div dir="ltr" style="text-align: left;" trbidi="on">
I uploaded my package for compression and decompression compressed RTF (aka LZFu or MELA) files to PyPI. This package is written in pure Python, and supports Python 2.7 and 3.x versions.<br />
<br />
Now it can be easily installed using pip command: <br />
<span style="font-family: "courier new" , "courier" , monospace;">pip install compressed_rtf</span><br />
<br />
The package is based on Rich Text Format (RTF) Compression Algorithm <a href="https://msdn.microsoft.com/en-us/library/cc463890(v=exchg.80).aspx" rel="nofollow">https://msdn.microsoft.com/en-us/library/cc463890(v=exchg.80).aspx</a><br />
<div>
<br /></div>
<div>
Sources are available on github:</div>
<div>
<a href="https://github.com/delimitry/compressed_rtf">https://github.com/delimitry/compressed_rtf</a></div>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-26025416511367068042018-04-19T01:04:00.002+06:002018-04-19T01:05:31.913+06:00RuCTF Finals 2018<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
This year I've decided to go to Yekaterinburg for the final of <a href="https://ructf.org/2018/">RuCTF 2018</a> to visit conference, watch the game (this time I was a spectator, because only students can play), see friends and and spent good time with a team.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjodOISJzy935tTdlCELWefxM2mGw54JYacdtKc6BdMMTJI9ZgrXP70-5dHojKkSAQT5BwmL59hVa_W1wgA0t5gZaixPrHELp7PybN90DkByPLD4LgvfBpeYsZMJGw_b39oVDnRoDNwrrk/s1600/plotinka.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="478" data-original-width="800" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjodOISJzy935tTdlCELWefxM2mGw54JYacdtKc6BdMMTJI9ZgrXP70-5dHojKkSAQT5BwmL59hVa_W1wgA0t5gZaixPrHELp7PybN90DkByPLD4LgvfBpeYsZMJGw_b39oVDnRoDNwrrk/s640/plotinka.jpg" width="640" /></a></div>
<div>
<br /></div>
<div>
We also visited local zoo.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOlwVmyEbnTqq93xpSeh9EAt8lhF7luvaUSZRMNveTQKT4zNjngsyIq7s2CLzqdJq7dMhWJdsNg8W_94YwXGHI4-rKPwk7DWqL-lxrLmY7KBUU8yF0zOELjpf1zXwszNpmeloytMwbboY/s1600/zoo.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="800" data-original-width="478" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOlwVmyEbnTqq93xpSeh9EAt8lhF7luvaUSZRMNveTQKT4zNjngsyIq7s2CLzqdJq7dMhWJdsNg8W_94YwXGHI4-rKPwk7DWqL-lxrLmY7KBUU8yF0zOELjpf1zXwszNpmeloytMwbboY/s640/zoo.jpg" width="382" /></a></div>
<br />
Conference flew by quickly, thanks to guys who made talks and thanks to guys from HWV. You can find some presentations here: <a href="http://live.ructf.org/">http://live.ructf.org/</a>.<br />
<br />
One more event on RuCTF is battles. Two people from different teams solve the task within 15 minutes. I've solved matrix task with my team mates :)<br />
<br />
RuCTF finals are held in Yeltsin Center with a huge display with game <a href="http://monitor.ructf.org/">monitor</a>.<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiibw47xIyx_wnRCE4Mur2YXV-FeICVnfQyGLZFxQKfCWP7kGTeMnjaTOESxm2HVfvH5wKy67cbYYbhnrbux4Cjq052YljDqo-Y27KA9gDaY_3s21RU3XWBqcj5yYv2-cTXtd1HlJ1OsMk/s1600/4.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="478" data-original-width="800" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiibw47xIyx_wnRCE4Mur2YXV-FeICVnfQyGLZFxQKfCWP7kGTeMnjaTOESxm2HVfvH5wKy67cbYYbhnrbux4Cjq052YljDqo-Y27KA9gDaY_3s21RU3XWBqcj5yYv2-cTXtd1HlJ1OsMk/s640/4.jpg" width="640" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcvxFX3xSIWg2cE-FvODNIQ5tp2X0UL2Pf3xyBMtjxZHULq8LAoh7xpBEnqG_QzgmwTJC_u7NgQf1SV4SpHIVqKyDEMm7DiVb8eY7l4PIWcSPgiW3cLJS40MtMhmrPg1pDMWlBDoqcQec/s1600/3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="478" data-original-width="800" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcvxFX3xSIWg2cE-FvODNIQ5tp2X0UL2Pf3xyBMtjxZHULq8LAoh7xpBEnqG_QzgmwTJC_u7NgQf1SV4SpHIVqKyDEMm7DiVb8eY7l4PIWcSPgiW3cLJS40MtMhmrPg1pDMWlBDoqcQec/s640/3.jpg" width="640" /></a></div>
<div>
<br /></div>
<div>
This year out team <a href="http://sibears.ru/">SiBears </a>finished in the 8th place (out of 26 teams in total, from 6 countries, 2 guest and 1 school team).</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjR2UZY22IFoKJ0g-a-0jImOmBMGKXWSd8J-1OESuW5d2Gk0lDRiCZQMmJiL_RlOg5bdbJBH1JC59SW_4Fu0ARiBIRmu2916bU5S_cYcEgfwG3a1AgZciC0LBcoQKCGhU7Ap0aexqHeBrs/s1600/ructf_2018_score.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="894" data-original-width="976" height="585" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjR2UZY22IFoKJ0g-a-0jImOmBMGKXWSd8J-1OESuW5d2Gk0lDRiCZQMmJiL_RlOg5bdbJBH1JC59SW_4Fu0ARiBIRmu2916bU5S_cYcEgfwG3a1AgZciC0LBcoQKCGhU7Ap0aexqHeBrs/s640/ructf_2018_score.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<div>
1st place: Bushwhackers from Moscow State University</div>
<div>
2nd place: c00kies@venice from Ca'Foscari University</div>
<div>
3rd place: saarsec from Saarland University</div>
</div>
<div>
<br /></div>
<div>
It was really hard game. More than 10 hours of stress. Out team and some other teams had an accidental power outage during the game :(</div>
<div>
By the way the game was delayed due to some problems with vulnerable image's disk.</div>
<div>
<br /></div>
<div>
After the game we made team photos and went to the after-party where guys played "Nalivaika" game, where it was required to answer security-related questions several rounds. Top 10 contestants drank Vodka after each round.</div>
<div>
<br /></div>
<div>
In the evening we gathered on the kitchen of our hotel to eat and discuss the game, team captain even has the strength to make some fixes in our exploit farm :)</div>
<div>
<br /></div>
<div>
Thanks RuCTF organizers, Hackerdom and all participants. </div>
<div>
RuCTF is friendship!</div>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-14362342401379749622018-03-19T00:58:00.002+06:002018-03-19T00:59:23.658+06:00Twin paradox and distributed systems<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
In the future we definetely will travel through the Universe. And the speed of the starships will be close to the speed of light, or even probably beyond it (as you can see in some movies). Therefore according to the <a href="https://en.wikipedia.org/wiki/Twin_paradox">twin paradox</a> all the clocks will have different time. Intergalactic distributed computer network will be really difficult, and eventually (I hope so) constistent.</div>
<div>
<br /></div>
<div>
So, I have some questions:</div>
<div>
How will the clocks in computers on the starships, stations, and planets be syncronized?</div>
<div>
There are special data structure called <a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">CRDT</a> (conflict-free replicated data type) but is also uses timestamps.</div>
<div>
Probably we need to find some source/invariant of global time (but how can we break the laws of physics (may be quantum physics can help here)), or find a way to make amendments base on straship acceleration/speed/traveled distance if it is possible.</div>
<div>
<br /></div>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-40021897844851303722018-03-17T09:40:00.000+06:002018-03-18T12:43:06.584+06:00iCTF 2018<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both;">
Today the historical event happened. The team SiBears I'm playing with won first place in iCTF 2018 CTF contest.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgakeBvoY3bbTW6OYQ2zTFK6weukcStpH4KS6NUAA92Y_A_7JLMphHwV3thKIZT1_7gQPZBg5ajbfJfCLQU6YHf8lQgM29cK8gwi59XSkp7mSe7OBcGmkm1PiESOpSHSOtDY0c8D8M6eTA/s1600/ictf2018_scoreboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="645" data-original-width="1294" height="319" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgakeBvoY3bbTW6OYQ2zTFK6weukcStpH4KS6NUAA92Y_A_7JLMphHwV3thKIZT1_7gQPZBg5ajbfJfCLQU6YHf8lQgM29cK8gwi59XSkp7mSe7OBcGmkm1PiESOpSHSOtDY0c8D8M6eTA/s640/ictf2018_scoreboard.png" width="640" /></a></div>
<br />
First time I've played iCTF in 2011. And it was hard but really interesting. And I'm very proud that out team has finally won, and finished at first place, because this is one of the first (if not the first) classic CTF competitions. Thanks to the UCSB iCTF team for such great event.</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-20803307465250914382018-03-16T03:26:00.000+06:002018-03-18T03:27:10.106+06:00Surely You're Joking, Mr. Feynman!<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I've finished reading the book "Surely You're Joking, Mr. Feynman!". This book was recommended in the book "A Mind For Numbers: How to Excel at Math and Science (Even If You Flunked Algebra)".<br />
I <span class="st">really</span> enjoyed reading it. The great and funny stories from the life of the great man.<br />
<div>
Recommend this book to everyone, it is really good.</div>
</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-9123928453218027602018-03-13T02:12:00.000+06:002018-10-17T01:52:45.458+06:00Гипотеза RUM<div dir="ltr" style="text-align: left;" trbidi="on">
Группа разработчиков из DASlab (Лаборатория систем обработки данных) Гарвардской школы инженерных и прикладных наук Джона А. Полсона (SEAS) разработали гипотезу RUM (Read, Update, Memory, то есть Чтение, Обновление/Запись и Память).<br />
<br />
Гипотеза RUM описывает повсеместный компромисс между оптимальным чтением, записью и используемой памятью у методов доступа к современным системам данных.<br />
<br />
При построении методов доступа к современным системам, каждый сталкивается с одними и теми же фундаментальными задачами и проектными решениями. <br />
Есть три величины, которые исследователи всегда стараются свести к минимуму:<br />
<ul style="text-align: left;">
<li>накладные расходы на чтение (R)</li>
<li>накладные расходы на обновление/запись (U)</li>
<li>накладные расходы на память (оперативная память, диск, и др.) (M)</li>
</ul>
три этих параметра (накладные расходы RUM) образуют треугольник:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0SxCGKrqwjCTrYT3OqpNG8U2_0e5xnAV6_e8J5Ln7mlBpp9ERj1gaImqxiL6V-0w489aesgbRPjqjoJqx0ZPzJAoQd16AGmNVBNLwI8aBx15b8oMp-nFDBuZd4E-_n_S8fpXNeb9ukBs/s1600/%25D0%2593%25D0%25B8%25D0%25BF%25D0%25BE%25D1%2582%25D0%25B5%25D0%25B7%25D0%25B0_RUM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="409" data-original-width="468" height="279" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0SxCGKrqwjCTrYT3OqpNG8U2_0e5xnAV6_e8J5Ln7mlBpp9ERj1gaImqxiL6V-0w489aesgbRPjqjoJqx0ZPzJAoQd16AGmNVBNLwI8aBx15b8oMp-nFDBuZd4E-_n_S8fpXNeb9ukBs/s320/%25D0%2593%25D0%25B8%25D0%25BF%25D0%25BE%25D1%2582%25D0%25B5%25D0%25B7%25D0%25B0_RUM.png" width="320" /></a></div>
<div style="text-align: center;">
<span style="font-size: x-small;">Накладные расходы RUM - на чтение, запись и используемую память.</span></div>
<br />
Гипотеза RUM:<br />
<i>разработка методов доступа, которые устанавливают верхнюю границу для двух (из трёх) накладныех расходов (</i><i>оверхедов</i><i>) RUM, приводит к жёсткой нижней границе для третьего </i><i>(</i><i>оверхеда</i><i>)</i><i>, которая не может быть дополнительно уменьшена.</i><br />
<br />
TLDR: <i>Чтение, Обновление/Запись, Память </i><span style="background-color: white; color: #545454; font-family: "arial" , sans-serif; font-size: x-small;">—</span><i> оптимизация двух за счет третьего.</i><br />
<div>
<br />
Накладные расходы на чтение выражаются в усилении чтения (read amplification), то есть отношении объёма фактически прочитанных данных к объему необходимых данных.<br />
Аналогично накладные расходы на обновление/запись выражаются в усилении записи (write amplification), а накладные расходы на память в усилении занимаемой памяти (space amplification).<br />
Здесь усилении записи (write amplification) <span style="background-color: white; color: #545454; font-family: "arial" , sans-serif; font-size: x-small;">—</span> это отношение объёма фактически записанных данных к реально изменённому, а усиление занимаемой памяти (space amplification) <span style="background-color: white; color: #545454; font-family: "arial" , sans-serif; font-size: x-small;">— </span>отношение объёма данных + объём вспомогательных данных таких как индексы и другие метаданные, к объёму самих данных.<br />
<br /></div>
<div>
При разработке методов доступа к системам, существует множество подходов, каждый из которых предназначен для конкретного случая использования. Например, чтобы минимизировать стоимость чтения данных, можно использовать индексы на основе хэш-таблиц или структуры данных с логарифмической сложностью, такие как Hash table, B-Tree, Skiplist, Trie и др.<br />
<br />
Структуры с оптимизацией записи <span style="background-color: white; color: #545454; font-family: "arial" , sans-serif; font-size: x-small;">—</span> уменьшают накладные расходы на запись, сохраняя обновления во вторичных структурах данных. Основная идея заключается в объединении обновлений с последующим применением к системе управления данными. Примеры: LSM-tree, Partitioned B-tree (PBT), алгоритм Materialized Sort-Merge (MaSM), алгоритм Stepped Merge и Positional Differential Tree (PDT), и др.<br />
<br />
Способы эффективного использования памяти предназначены для уменьшения накладных расходов на хранение данных. К ним относятся методы сжатия и индексы с потерями, такие как Фильтр Блума, HyperLogLog, Count-min sketch, Bitmap, Zonemap, Sparse индексы, и др.</div>
<br />
<b>Ссылки:</b><br />
[1] <a href="http://daslab.seas.harvard.edu/rum-conjecture/">The RUM Conjecture <span style="font-family: monospace; white-space: pre-wrap;">-</span></a><a href="http://daslab.seas.harvard.edu/rum-conjecture/"> </a>DASlab @ Harvard University<br />
[2] Manos Athanassoulis, Michael S. Kester, Lukas M. Maas, et al.: “<a href="http://openproceedings.org/2016/conf/edbt/paper-12.pdf">Designing Access Methods: The RUM Conjecture</a>,” at 19th International Conference on Extending Database Technology (EDBT), March 2016. doi:10.5441/002/edbt.2016.42<br />
<br /></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-68148800641847564482018-02-25T20:46:00.000+06:002018-02-25T20:46:39.304+06:00A Mind For Numbers by Barbara Oakley<div dir="ltr" style="text-align: left;" trbidi="on">
I've finished a book "A Mind For Numbers: How to Excel at Math and Science (Even If You Flunked Algebra)"<br />
Good book with good advices and techniques of how to effectively learn, some of them I have used many times :)<br />
I wish I have read this book earlier. Highly recommend this book to students and pupils.</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-62530051774170532582018-02-07T05:17:00.003+06:002018-02-07T05:18:57.111+06:00Buzzword poem generator algorithm<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Conditions of the problem</b></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Given: a list of buzzwords.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Problem: generate poem from these buzzwords.<br /><br />We also need to know:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">1) Result poem lines number.<br />2) Rhyme scheme.<br />3) Syllables in each line.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">4) Minimum number of words in each line.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">also:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">5) Somehow know the number of syllables in each buzzword.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">6) Somehow get the list of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">rhyming </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">buzzwords</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">.</span><br />
<br />
<b><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Algorithm</span></b><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">1) Knowing the total number of syllables in each line </span><span style="background-color: white; color: #333333; font-family: "georgia" , serif;">—</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> generate all combinations of sums (compositions)</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> from all available number of syllables.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">E.g. </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">we have buzzwords with the number of syllables 1, 2 and 3, and the</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">total number of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">syllables in line (i.e. sum of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">syllables</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">) equal 3. C</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">ompositions of 3 are:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">1 </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">+ 1 + 1</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">1 </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">+ 2</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">2 </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">+ 1</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">3</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If we assume that we don't have buzzwords with </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">number of syllables 2 then only two possible combinations can be: 1 + 1 + 1 and 3.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">NB: Each positive integer n has 2 ** (n - 1) distinct compositions.</span><br />
<div>
<br /></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">2) Knowing the number of syllables in each line, generate the base (I call it so) for future poem. The base is generated from random compositions for each </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">number of syllables</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">E.g. for two line poem with 3 and 4 syllables, the possible base can be the next:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[2, 1]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[1, 2, 1]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">or</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[2, 1]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[2, 2] </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">The poem base can be also limited to the minimum number of words per line. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">For the example above with the m</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">inimum number of words equal 3, possible base can be the next:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[1, 1, 1]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[1, 1, 2]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">or (notice that the first line has only one possible option):</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[1, 1, 1]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[1, 1, 1, 1]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">3) Knowing the rhyme scheme it is possible to get the last number of syllables for each line from poem base and group these numbers by rhyme scheme letters, thereby simplify the search of rhyme buzzwords.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">The last </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">number of syllables for each line </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">are taken because only the last words form a rhyme.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">For example the rhyme scheme is ABA, and the poem base is:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[1, 2, 1]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[1, 1, 2]</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[3, 2]<br />Last numbers of syllables for each line are 1, 2 and 2.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">1 A</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">2 B</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">2 A</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Group them by rhyme scheme letters: <br />A: [1, 2], B: [2]<br /><br />It is required to find all buzzwords that are rhyme, and have 1 and 2 syllables, then randomly select one</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">from the list of found variants for A letter.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">And find only one random buzzword with 2 syllables for B </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">letter</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">It is important that after using a buzzword, it must be removed from the list of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">buzzwords to </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">avoid duplication.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If the rhymes was not found it is required to indicate that, for example by returning empty result.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">4) </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">To this step we already have poem base, and rhyme scheme letters </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">mapped to the found buzzwords. </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">In case of empty </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">mapping of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">rhyme scheme </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">letters to buzzwords</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> — go to step 2 and try again.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If the </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">mapping of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">rhyme scheme </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">letters to buzzwords</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> isn't empty, start filling the poem base with buzzwords according to the number of syllables. The filling of each poem base </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">line</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">must be performed until the penultimate number of syllables, because the last ones were already found in previous step and saved in the </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">mapping</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">rhyme scheme </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">letters to buzzwords</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">As in the previous step, it is important that after using a buzzword, it must be removed from the list of </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">buzzwords to </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">avoid duplication.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If the filling of the poem was not succeed </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">— go to step 2 and try again.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If the number of attempts is too big, stop trying and display the error message.<br /><br /><b>Enhancements</b></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Support the poem metrical feet, to generate more smooth poems.<br /><br />This algorithm was used in my project: <a href="https://github.com/delimitry/buzzword_poem_generator">https://github.com/delimitry/buzzword_poem_generator</a></span></div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0tag:blogger.com,1999:blog-1507940912782544027.post-10967248449876952382018-02-06T04:58:00.004+06:002018-02-06T04:58:54.713+06:00Buzzword poem generator<div dir="ltr" style="text-align: left;" trbidi="on">
Last week I've created a new project on my <a href="https://github.com/delimitry/">github</a> — a tool for the generation of poems from the buzzwords. I called it <a href="https://github.com/delimitry/buzzword_poem_generator">Buzzword poem generator</a>. As usual, the tool is written in pure Python and supports 2.7 and, of course, Python 3.x.<br /><br />
The project was inspired by recent tweet from Lena Hall (<a href="https://twitter.com/lenadroid">@lenadroid </a>on twitter): <a href="https://twitter.com/lenadroid/status/957054198872293378" rel="nofollow">https://twitter.com/lenadroid/status/957054198872293378</a><br />
<br />My tool generates poems based on the list of buzzwords. It is possible to pass the parameters of the rhyme scheme and number of syllables in each line. For example for the rhyme scheme ABAB, and 7 syllables per each line, the next poem was generated:<br /><b style="font-family: "Courier New", Courier, monospace;"><br />$ python buzzword_poem_generator.py -r ABAB -s 7 7 7 7</b><br />
<span style="font-family: Courier New, Courier, monospace;"><b>Memcached Vault Impala Spark</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>Redshift Chef Celery Swarm</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>Haskell Rust Zookeeper Splunk</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>Python Flink Kinesis Storm</b></span><br />
<div>
<br /></div>
Use it with pleasure :)</div>
Delimitryhttp://www.blogger.com/profile/12169837382010969209noreply@blogger.com0