Learn Web Development with Python 1789953294, 9781789953299

Citation preview

---

Learn Web Development with Python

Get hands-on with Python Programming and Django web development

Fabrizio Romano Gastón C. Hillar Arun Ravindran

BIRMINGHAM - MUMBAI

Learn Web Development with Python Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: December 2018 Production reference: 1201218 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78995-329-9 www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content

Packt.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors About the authors Fabrizio Romano was born in Italy in 1975. He holds a master's degree in computer science engineering from the University of Padova. He is also a certified scrum master, Reiki master and teacher, and a member of CNHC. He moved to London in 2011 to work for companies such as Glasses Direct, and TBG/Sprinklr. He now works at Sohonet as a Principal Engineer/Team Lead. He has given talks on Teaching Python and TDD at two editions of EuroPython, and at Skillsmatter and ProgSCon, in London. I'm grateful to all those who helped me create this book. Special thanks to Dr. Naomi Ceder for writing the foreword to this edition, and to Heinrich Kruger and Julio Trigo for reviewing this volume. To my friends and family, who love me and support me every day, thank you. And to Petra Lange, for always being so lovely to me, thank you.

Gaston C. Hillar is Italian and has been working with computers since he was eight years old. Gaston has a bachelor's degree in computer science (graduated with honors) and an MBA. He is an independent consultant, a freelance author, and a speaker. He has been a senior contributing editor at Dr. Dobb's and has written more than a hundred articles on software development topics. He has received the prestigious Intel® Black Belt Software Developer award eight times. He lives with his wife, Vanesa, and his two sons, Kevin and Brandon. At the time of writing this book, I was fortunate to work with an excellent team at Packt, whose contributions vastly improved the presentation of this book. Reshma Raman allowed me to provide her ideas to write a book dedicated to RESTful Web Services development with Django and Python, and I jumped into the exciting project. Aditi Gour helped me realize my vision for this book and provided many sensible suggestions regarding the text, the format, and the flow. The reader will notice her great work. It′s been great working with Reshma on another project and I can't wait to work with Reshma and Aditi again. I would like to thank my technical reviewers and proofreaders, for their thorough reviews and insightful comments. I was able to incorporate some of the knowledge and wisdom they have gained in their many years in the software development industry. This book was possible because they gave valuable feedback. The entire process of writing a book requires a huge number of lonely hours. I wouldn't be able to write an entire book without dedicating some time to play soccer against my sons, Kevin and Brandon, and my nephew, Nicolas. Of course, I never won a match. However, I did score a few goals. Of course, I'm talking about real-life soccer, but I must also add virtual soccer when the weather didn't allow us to kick a real-life ball.

Arun Ravindran is an avid speaker and blogger who has been tinkering with Django since 2007 for projects ranging from intranet applications to social networks. He is a long-time open source enthusiast and Python developer. His articles and screencasts have been invaluable to the rapidly growing Django community. He is currently a developer member of the Django Software Foundation. Arun is also a movie buff and loves graphic novels and comics. To my wife, Vidya, for her constant support and encouragement. To my daughter, Kavya, who showed understanding beyond her age when her dad was devoted to writing. To my son, Nihar, who is almost as old as the first edition of this book. A big thanks to all the wonderful people at Packt Publishing who helped in the creation of the first and second editions of this book. Truly appreciate the honest reviews the wonderful technical reviewer. Sincere thanks to the author Anil Menon for his inputs on the SuperBook storyline. I express my unending appreciation of the entire Django and Python community for being open, friendly and incredibly collaborative. Without their hard work and generosity, we would not have the great tools and knowledge that we depend on everyday. Last but not the least, special thanks to my family and friends who have always been there to support me.

About the reviewers Heinrich Kruger was born in South Africa in 1981. He obtained a bachelor's degree with honors from the University of the Witwatersrand in South Africa in 2005 and a master's degree in computer science from Utrecht University in the Netherlands in 2008. He worked as a research assistant at Utrecht University from 2009 until 2013 and has been working as a professional software developer developer since 2014. He has been using Python for personal and projects and in his studies since 2004, and professionally since 2014. Julio Vicente Trigo Guijarro is a computer science engineer with over a decade of experience in software development. He completed his studies at the University of Alicante, Spain, in 2007 and moved to London in 2010. He has been using Python since 2012 and currently works as a senior software developer and team lead at Sohonet, developing real-time collaboration applications for the media industry. He is also a certified ScrumMaster and was one of the technical reviewers of the first edition of this book. I would like to thank my parents for their love, good advice, and continuous support. I would also like to thank all the friends I have met along the way, who enriched my life, for keeping up my motivation, and make me progress. Norbert Mate is a web developer who started his career back in 2008. His first programming language as a professional web development was PHP, and then he moved on to JavaScript/node.js and Python/Django/Django REST framework. He is passionate about software architecture, design patterns, and clean code. Antoni Aloy is a computer engineer graduated from the Universitat Oberta de Catalunya (UOC). He has been working with Python since 1999 and with Django since its early releases. In 2009, he founded APSL (apsl.net), a development and IT company based in Mallorca (Spain), in which Python and Django are the backbone of the software development department. He is also a founding member of the Python España Association and promotes the use of Python and Django through workshops and articles. I would like to thank my family, coworkers, and the amazing Python and Django community.

Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents Preface Chapter 1: A Gentle Introduction to Python A proper introduction Enter the Python About Python Portability Coherence Developer productivity An extensive library Software quality Software integration Satisfaction and enjoyment

What are the drawbacks? Who is using Python today? Setting up the environment Python 2 versus Python 3

Installing Python

Setting up the Python interpreter About virtualenv Your first virtual environment Your friend, the console

How you can run a Python program Running Python scripts Running the Python interactive shell Running Python as a service Running Python as a GUI application

How is Python code organized?

How do we use modules and packages?

Python's execution model Names and namespaces Scopes Objects and classes

Guidelines on how to write good code The Python culture A note on IDEs Summary Chapter 2: Built-in Data Types Everything is an object

1 10 12 14 14 14 15 15 15 16 16 16 16 17 18 18 19 20 22 23 26 26 26 27 28 29 29 31 34 34 36 39 42 43 44 45 46 47

Table of Contents

Mutable or immutable? That is the question Numbers Integers Booleans Real numbers Complex numbers Fractions and decimals

Immutable sequences Strings and bytes

Encoding and decoding strings Indexing and slicing strings String formatting

Tuples

Mutable sequences Lists Byte arrays

Set types Mapping types – dictionaries The collections module namedtuple defaultdict ChainMap

Enums Final considerations

Small values caching How to choose data structures About indexing and slicing About the names

Summary Chapter 3: Iterating and Making Decisions Conditional programming A specialized else – elif The ternary operator

Looping

The for loop

Iterating over a range Iterating over a sequence

Iterators and iterables Iterating over multiple sequences The while loop The break and continue statements A special else clause

Putting all this together A prime generator

[ ii ]

48 49 49 51 52 54 54 56 56 57 57 58 59 61 61 65 66 68 72 73 74 75 76 77 78 78 80 81 82 84 85 86 88 89 89 90 91 92 94 96 99 101 103 103

Table of Contents

Applying discounts

A quick peek at the itertools module

Infinite iterators Iterators terminating on the shortest input sequence Combinatoric generators

Summary Chapter 4: Functions, the Building Blocks of Code Why use functions? Reducing code duplication Splitting a complex task Hiding implementation details Improving readability Improving traceability

Scopes and name resolution

The global and nonlocal statements

Input parameters

Argument-passing Assignment to argument names doesn't affect the caller Changing a mutable affects the caller How to specify input parameters Positional arguments Keyword arguments and default values Variable positional arguments Variable keyword arguments Keyword-only arguments Combining input parameters Additional unpacking generalizations Avoid the trap! Mutable defaults

Return values

Returning multiple values

A few useful tips Recursive functions Anonymous functions Function attributes Built-in functions One final example Documenting your code Importing objects Relative imports

Summary Chapter 5: Saving Time and Memory The map, zip, and filter functions map zip

[ iii ]

105 108 109 110 111 111 113 114 115 115 116 117 118 119 120 122 122 124 124 125 125 126 127 128 130 130 132 133 134 136 137 138 139 140 141 142 143 144 146 146 148 150 150 153

Table of Contents

filter

Comprehensions

Nested comprehensions Filtering a comprehension dict comprehensions set comprehensions

Generators

Generator functions Going beyond next The yield from expression Generator expressions

Some performance considerations Don't overdo comprehensions and generators Name localization Generation behavior in built-ins One last example Summary Chapter 6: OOP, Decorators, and Iterators Decorators A decorator factory

Object-oriented programming (OOP)

The simplest Python class Class and object namespaces Attribute shadowing Me, myself, and I – using the self variable Initializing an instance OOP is about code reuse Inheritance and composition

Accessing a base class Multiple inheritance

Method resolution order

Class and static methods Static methods Class methods

Private methods and name mangling The property decorator Operator overloading Polymorphism – a brief overview Data classes

Writing a custom iterator Summary Chapter 7: Files and Data Persistence Working with files and directories Opening files

[ iv ]

154 155 156 157 159 160 161 161 164 168 169 172 175 179 180 181 183 184 184 190 192 193 194 195 196 197 198 198 203 206 209 211 211 213 215 218 220 221 221 222 224 225 226 226

Table of Contents

Using a context manager to open a file

Reading and writing to a file

Reading and writing in binary mode Protecting against overriding an existing file

Checking for file and directory existence Manipulating files and directories Manipulating pathnames

Temporary files and directories Directory content File and directory compression

Data interchange formats Working with JSON

Custom encoding/decoding with JSON

IO, streams, and requests

Using an in-memory stream Making HTTP requests

Persisting data on disk

Serializing data with pickle Saving data with shelve Saving data to a database

Summary Chapter 8: Testing, Profiling, and Dealing with Exceptions Testing your application The anatomy of a test Testing guidelines Unit testing

Writing a unit test Mock objects and patching Assertions

Testing a CSV generator

Boundaries and granularity Testing the export function Final considerations

Test-driven development Exceptions Profiling Python When to profile?

Summary Chapter 9: Concurrent Execution Concurrency versus parallelism Threads and processes – an overview Quick anatomy of a thread Killing threads Context-switching

The Global Interpreter Lock

[v]

228 228 229 230 230 231 233 234 235 236 236 237 240 244 244 246 249 249 251 253 260 261 262 264 265 267 267 269 269 270 280 280 283 285 287 293 295 297 298 299 300 300 301 302 302

Table of Contents

Race conditions and deadlocks Race conditions

Scenario A – race condition not happening Scenario B – race condition happening

Locks to the rescue

Scenario C – using a lock

Deadlocks

Quick anatomy of a process Properties of a process

Multithreading or multiprocessing?

Concurrent execution in Python Starting a thread Starting a process Stopping threads and processes Stopping a process

Spawning multiple threads Dealing with race conditions A thread's local data Thread and process communication

Thread communication Sending events Inter-process communication with queues

Thread and process pools Using a process to add a timeout to a function

Case examples

Example one – concurrent mergesort Single-thread mergesort Single-thread multipart mergesort Multithreaded mergesort Multiprocess mergesort

Example two – batch sudoku-solver

What is Sudoku? Implementing a sudoku-solver in Python Solving sudoku with multiprocessing

Example three – downloading random pictures Downloading random pictures with asyncio

Summary Chapter 10: Debugging and Troubleshooting Debugging techniques Debugging with print Debugging with a custom function Inspecting the traceback Using the Python debugger Inspecting log files Other techniques Profiling Assertions

[ vi ]

303 304 304 304 305 305 305 306 307 308 309 309 312 312 313 314 315 317 318 319 320 321 322 325 328 328 329 330 331 332 334 334 335 340 343 345 349 350 352 352 352 354 357 360 362 363 363

Table of Contents

Where to find information

Troubleshooting guidelines Using console editors Where to inspect Using tests to debug Monitoring

Summary Chapter 11: Installing the Required Software and Tools Creating a virtual environment with Python 3.x and PEP 405 Understanding the directory structure for a virtual environment Activating the virtual environment Deactivating the virtual environment

Installing Django and Django REST frameworks in an isolated environment Creating an app with Django Understanding Django folders, files, and configurations

Installing tools

Installing Curl Installing HTTPie Installing the Postman REST client Installing Stoplight Installing iCurlHTTP

Test your knowledge Summary Chapter 12: Working with Models, Migrations, Serialization, and Deserialization Defining the requirements for our first RESTful Web Service Creating our first model Running our initial migration Understanding migrations

Analyzing the database

Understanding the table generated by Django

Controlling, serialization, and deserialization Working with the Django shell and diving deeply into serialization and deserialization Test your knowledge Summary Chapter 13: Creating API Views Creating Django views combined with serializer classes Understanding CRUD operations with Django views and the request methods Routing URLs to Django views and functions [ vii ]

363 364 364 364 365 365 366 367 368 370 371 374 375 376 378 380 380 383 384 385 386 388 389 390 391 394 395 396 398 400 401 403 410 411 412 413 415 418

Table of Contents

Launching Django's development server

Making HTTP GET requests that target a collection of instances Making HTTP GET requests that target a single instance Making HTTP POST requests Making HTTP PUT requests Making HTTP DELETE requests Making HTTP GET requests with Postman

Making HTTP POST requests with Postman Test your knowledge Summary Chapter 14: Using Generalized Behavior from the APIView Class Taking advantage of model serializers Understanding accepted and returned content types Making unsupported HTTP OPTIONS requests with commandline tools Understanding decorators that work as wrappers Using decorators to enable different parsers and renderers Taking advantage of content negotiation classes Making supported HTTP OPTIONS requests with commandline tools Working with different content types Sending HTTP requests with unsupported HTTP verbs Test your knowledge Summary Chapter 15: Understanding and Customizing the Browsable API Feature Understanding the possibility of rendering text/HTML content Using a web browser to work with our web service Making HTTP GET requests with the browsable API Making HTTP POST requests with the browsable API Making HTTP PUT requests with the browsable API Making HTTP OPTIONS requests with the browsable API Making HTTP DELETE requests with the browsable API Test your knowledge Summary Chapter 16: Using Constraints, Filtering, Searching, Ordering, and Pagination Browsing the API with resources and relationships Defining unique constraints Working with unique constraints Understanding pagination [ viii ]

419 420 426 428 429 432 433 435 438 439 440 441 443 444 447 449 451 453 455 457 458 459 460 460 463 465 468 471 474 476 479 480 481 482 486 491 494

Table of Contents

Configuring pagination classes Making requests that paginate results Working with customized pagination classes Making requests that use customized paginated results Configuring filter backend classes Adding filtering, searching, and ordering Working with different types of Django filters Making requests that filter results Composing requests that filter and order results Making requests that perform starts with searches Using the browsable API to test pagination, filtering, searching, and ordering Test your knowledge Summary Chapter 17: Securing the API with Authentication and Permissions Understanding authentication and permissions in Django, the Django REST framework, and RESTful Web Services Learning about the authentication classes Including security and permissions-related data to models Working with object-level permissions via customized permission classes Saving information about users that make requests Setting permission policies Creating the superuser for Django Creating a user for Django Making authenticated requests Making authenticated HTTP PATCH requests with Postman Browsing the secured API with the required authentication Working with token-based authentication Generating and using tokens Test your knowledge Summary Chapter 18: Applying Throttling Rules and Versioning Management Understanding the importance of throttling rules Learning the purpose of the different throttling classes in the Django REST framework Configuring throttling policies in the Django REST framework Running tests to check that throttling policies work as expected [ ix ]

495 498 503 505 506 509 512 516 517 521 522 526 527 528 529 530 531 535 537 538 539 544 545 549 552 555 558 561 562 563 564 564 568 571

Table of Contents

Understanding versioning classes Configuring a versioning scheme Running tests to check that versioning works as expected Test your knowledge Summary Chapter 19: Automating Tests Getting ready for unit testing with pytest Writing unit tests for a RESTful Web Service Discovering and running unit tests with pytest Writing new unit tests to improve the tests' code coverage Running unit tests again with pytest Test your knowledge Summary Chapter 20: Solutions Chapter 11: Installing the Required Software and Tools Chapter 12: Working with Models, Migrations, Serialization, and Deserialization Chapter 13: Creating API Views Chapter 14: Using Generalized Behavior from the APIView Class Chapter 15: Understanding and Customizing the Browsable API Feature Chapter 16: Using Constraints, Filtering, Searching, Ordering, and Pagination Chapter 17: Securing the API with Authentication and Permissions Chapter 18: Applying Throttling Rules and Versioning Management Chapter 19: Automating Tests Chapter 21: Templates Understanding Django's template language features Variables Attributes Filters Tags Philosophy – don't invent a programming language

Jinja2 Organizing templates How templates work Using Bootstrap

But they all look the same!

[x]

576 578 582 588 589 590 590 593 599 603 608 610 611 612 612 612 613 613 613 614 614 614 615 616 616 617 617 618 618 619 620 621 622 625 626

Table of Contents

Lightweight alternatives

Template patterns

Pattern — template inheritance tree Problem details Solution details

Pattern — the active link Problem details Solution details

A template-only solution Custom tags

Summary Chapter 22: Admin Interface Using the admin interface Enhancing models for the admin Not everyone should be an admin

Admin interface customizations

Changing the heading Changing the base and stylesheets

Adding a rich-text editor for WYSIWYG editing

Bootstrap-themed admin Complete overhauls

Protecting the admin

Pattern – feature flags Problem details Solution details

Summary Chapter 23: Forms How forms work

Forms in Django Why does data need cleaning?

Displaying forms Time to be crisp

Understanding CSRF Form processing with class-based views Form patterns Pattern – dynamic form generation Problem details Solution details

Pattern – user-based forms Problem details Solution details

Pattern – multiple form actions per view Problem details Solution details

Separate views for separate actions

[ xi ]

627 628 628 628 629 631 631 631 631 632 633 634 634 637 641 642 642 642 643 644 644 645 646 646 646 648 649 649 650 654 655 656 657 658 659 659 659 660 661 661 662 662 663 663 663

Table of Contents Same view for separate actions

Pattern – CRUD views Problem details Solution details

Summary Chapter 24: Security Cross-site scripting

Why are your cookies valuable? How Django helps Where Django might not help

Cross-site request forgery

How Django helps Where Django might not help

SQL injection

How Django helps Where Django might not help

Clickjacking

How Django helps

Shell injection

How Django helps And the web attacks are unending

A handy security checklist Summary Chapter 25: Working Asynchronously Why asynchronous? Pitfalls of asynchronous code

Asynchronous patterns

Endpoint callback pattern Publish-subscribe pattern Polling pattern

Asynchronous solutions for Django Working with Celery

How Celery works Celery best practices

Handling failure Idempotent tasks Avoid writing to shared or global state Database updates without race conditions Avoid passing complex objects to tasks

Understanding asyncio

asyncio versus threads The classic web-scraper example Synchronous web-scraping Asynchronous web-scraping

Concurrency is not parallelism

Entering Channels

[ xii ]

663 665 665 666 668 669 669 671 672 673 673 674 674 674 675 675 676 677 677 677 678 681 683 684 684 685 686 686 687 687 688 688 689 690 691 692 693 694 695 695 696 697 697 698 701 701

Table of Contents

Listening to notifications with WebSockets Differences from Celery

Summary Chapter 26: Creating APIs RESTful API API design

Versioning

Django Rest framework

Improving the Public Posts API Hiding the IDs

API patterns

Pattern – human browsable interface Problem details Solution details

Pattern – Infinite Scrolling Problem details Solution details

Summary Chapter 27: Production-Ready The production environment

703 705 706 707 707 709 709 710 710 713 714 714 714 715 716 717 717 718 719 719 720 721 721 722 723 723 724 724 725 726 726 727 727 728 729 730 731 731 732 733 733 733 734 739

Choosing a web stack Components of a stack

Virtual machines or Docker Microservices

Hosting

Platform as a service Virtual private servers Serverless Other hosting approaches

Deployment tools Fabric

Typical deployment steps

Configuration management

Monitoring Improving Performance Frontend performance Backend performance Templates Database Caching

Cached session backend Caching frameworks Caching patterns

Summary

[ xiii ]

Table of Contents

Other Books You May Enjoy

740

Index

743

[ xiv ]

Preface If you want to develop complete Python web apps with Django, this Learning Path is for you. It will walk you through Python programming techniques and guide you in implementing them when creating 4 professional Django projects, teaching you how to solve common problems and develop RESTful web services with Django and Python. You will learn how to build a blog application, a social image bookmarking website, an online shop, and an e-learning platform. Learn Web Development with Python will get you started with Python programming techniques, show you how to enhance your applications with AJAX, create RESTful APIs, and set up a production environment for your Django projects. Last but not least, you’ll learn the best practices for creating of real-world applications. By the end of this Learning Path, you will have a full understanding of how Django works and how use it to build web applications from scratch. This Learning Path includes content from the following Packt products: Learn Python Programming by Fabrizio Romano Django RESTful Web Services by Gastón C. Hillar Django Design Patterns and Best Practices by Arun Ravindran

Who this book is for If you have little experience in coding or Python and want to learn how to build fullfledged web apps, this Learning Path is for you. No prior experience with RESTful web services, Python, or Django is required, but basic Python programming experience is needed to understand the concepts covered.

What this book covers Chapter 1, A Gentle Introduction to Python, introduces you to fundamental

programming concepts. It guides you through getting Python up and running on your computer and introduces you to some of its constructs.

Preface Chapter 2, Built-in Data Types, introduces you to Python built-in data types. Python

has a very rich set of native data types, and this chapter will give you a description and a short example for each of them.

Chapter 3, Iterating and Making Decisions, teaches you how to control the flow of your

code by inspecting conditions, applying logic, and performing loops.

Chapter 4, Functions, the Building Blocks of Code, teaches you how to write functions.

Functions are the keys to reusing code, to reducing debugging time, and, in general, to writing better code. Chapter 5, Saving Time and Memory, introduces you to the functional aspects of

Python programming. This chapter teaches you how to write comprehensions and generators, which are powerful tools that you can use to speed up your code and save memory. Chapter 6, OOP, Decorators, and Iterators, teaches you the basics of object-oriented

programming with Python. It shows you the key concepts and all the potentials of this paradigm. It also shows you one of the most beloved characteristics of Python: decorators. Finally, it also covers the concept of iterators. Chapter 7, Files and Data Persistence, teaches you how to deal with files, streams, data

interchange formats, and databases, among other things.

Chapter 8, Testing, Profiling, and Dealing with Exceptions, teaches you how to make

your code more robust, fast, and stable using techniques such as testing and profiling. It also formally defines the concept of exceptions. Chapter 9, Concurrent Execution, is a challenging chapter that describes how to do

many things at the same time. It provides an introduction to the theoretical aspects of this subject and then presents three nice exercises that are developed with different techniques, thereby enabling the reader to understand the differences between the paradigms presented. Chapter 10, Debugging and Troubleshooting, shows you the main methods for

debugging your code and some examples on how to apply them.

[2]

Preface Chapter 11, Installing the Required Software and Tools, shows how to get started in our

journey toward creating RESTful Web Services with Python and its most popular web framework—Django. We will install and configure the environments, the software, and the tools required to create RESTful Web Services with Django and Django REST framework. We will learn the necessary steps in Linux, macOS, and Windows. We will create our first app with Django, we will take a first look at the Django folders, files, and configurations, and we will make the necessary changes to activate Django REST framework. In addition, we will introduce and install command-line and GUI tools that we will use to interact with the RESTful Web Services that we will design, code, and test in the forthcoming chapters. Chapter 12, Working with Models, Migrations, Serialization, and Deserialization,

describes how to design a RESTful Web Service to interact with a simple SQLite database and perform CRUD operations with toys. We will define the requirements for our web service, and we will understand the tasks performed by each HTTP method and the different scopes. We will create a model to represent and persist toys and execute migrations in Django to create the required tables in the database. We will analyze the tables and learn how to manage the serialization of toy instances into JSON representations with Django REST framework and the reverse process. Chapter 13, Creating API Views, is about executing the first version of a simple Django

RESTful Web Service that interacts with a SQLite database. We will write API views to process diverse HTTP requests on a collection of toys and on a specific toy. We will work with the following HTTP verbs: GET, POST, and PUT. We will configure the URL patterns list to route URLs to views. We will start the Django development server and use command-line tools (curl and HTTPie) to compose and send diverse HTTP requests to our RESTful Web Service. We will learn how HTTP requests are processed in Django and our code. In addition, we will work with Postman, a GUI tool, to compose and send other HTTP requests to our RESTful Web Service. Chapter 14, Using Generalized Behavior from the APIView Class, presents different ways

to improve our simple Django RESTful Web Service. We will take advantage of many features included in the Django REST framework to remove duplicate code and add many features for the web service. We will use model serializers, understand the different accepted and returned content types, and the importance of providing accurate responses to the HTTP OPTIONS requests. We will make the necessary changes to the existing code to enable diverse parsers and renderers. We will learn how things work under the hoods in Django REST framework. We will work with different content types and note how the RESTful Web Service improves compared to its previous versions.

[3]

Preface Chapter 15, Understanding and Customizing the Browsable API Feature, explains how to

use one of the additional features that Django REST framework adds to our RESTful Web Service—the browsable API. We will use a web browser to work with our first web service built with Django. We will learn to make HTTP GET, POST, PUT, OPTIONS, and DELETE requests with the browsable API. We will be able to easily test CRUD operations with a web browser. The browsable API will allow us to easily interact with our RESTful Web Service. Chapter 16, Using Constraints, Filtering, Searching, Ordering, and Pagination, describes

the usage of the browsable API feature to navigate through the API with resources and relationships. We will add unique constraints to improve the consistency of the models in our RESTful Web Service. We will understand the importance of paginating results and configure and test a global limit/offset pagination scheme with Django REST framework. Then, we will create our own customized pagination class to ensure that requests won't be able to require a huge number of elements on a single page. We will configure filter backend classes and incorporate code into the models to add filtering, searching, and ordering capabilities to the class-based views. We will create a customized filter and make requests to filter, search, and order results. Finally, we will use the browsable API to test pagination, filtering, and ordering. Chapter 17, Securing the API with Authentication and Permissions, presents the

differences between authentication and permissions in Django, Django REST framework, and RESTful Web Services. We will analyze the authentication classes included in Django REST framework out of the box. We will follow the steps needed to provide security- and permissions-related data to models. We will work with object-level permissions via customized permission classes and save information about users who make requests. We will configure permission policies and compose and send authenticated requests to understand how the permission policies work. We will use command-line tools and GUI tools to compose and send authenticated requests. We will browse the secure RESTful Web Service with the browsable API feature and work with a simple token-based authentication provided by Django REST framework to understand another way of authenticating requests.

[4]

Preface Chapter 18, Applying Throttling Rules and Versioning Management, focuses on the

importance of throttling rules and how we can combine them with authentication and permissions in Django, Django REST framework, and RESTful Web Services. We will analyze the throttling classes included in Django REST framework out of the box. We will follow the necessary steps to configure many throttling policies in Django REST framework. We will work with global and scope-related settings. Then, we will use command-line tools to compose and send many requests to test how the throttling rules work. We will understand versioning classes and we will configure a URL path versioning scheme to allow us to work with two versions of our RESTful Web Service. We will use command-line tools and the Browsable API to understand the differences between the two versions. Chapter 19, Automating Tests, shows how to automate tests for our RESTful Web

Services developed with Django and Django REST framework. We will use different packages, tools, and configurations to perform tests. We will write the first round of unit tests for our RESTful Web Service, run them, and measure tests code coverage. Then, we will analyze tests code coverage reports and write new unit tests to improve the test code coverage. We will understand the new tests code coverage reports and learn the benefits of a good test code coverage. Chapter 20, Solutions, the right answers for the Test Your Knowledge sections of each

chapter are included in the appendix.

Chapter 21, Templates, walks us through Django template language constructs,

explaining its design choices, suggests how to organize template files, introduces handy template patterns, and points to several ways Bootstrap can be integrated and customized. Chapter 22, Admin Interface, focuses on how to use Django's brilliant out-of-the box

admin interface more effectively and several ways to customize it, from enhancing the models to toggling feature flags.

[5]

Preface Chapter 23, Forms, illustrates the often confusing form workflow, different ways of

rendering forms, improving a form's appearance using crispy forms, and various applied form patterns.

Chapter 24, Working Asynchronously, tours various asynchronous solutions for the

Django developer, from the feature-rich Celery task queues, Python 3's asyncio, to the brand new Channels, and compares them for you. Chapter 25, Creating APIs, explains RESTful API design concepts with practical

advice on topics such as versioning, error handling, and design patterns using the Django REST framework. Chapter 26, Security, familiarizes you with various web security threats and their

counter measures, specifically looking at how Django can protect you. Finally, a handy security checklist reminds you of the commonly overlooked areas. Chapter 27, Production-Ready, is a crash course in deploying a public-facing

application beginning with choosing your webstack, understanding hosting options, and walking through a typical deployment process. We go into the details of monitoring and performance at this stage.

To get the most out of this book You will just need a computer (PC or Mac) and internet connectivity to start with. Then, ensure that the following are installed: Python 3.4 or later Django 2 or later (will be covered in installation instructions) Text Editor (or a Python IDE) Web browser (the latest version, please)

Download the example code files You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.

[6]

Preface

You can download the code files by following these steps: 1. 2. 3. 4.

Log in or register at www.packt.com. Select the SUPPORT tab. Click on Code Downloads & Errata. Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of: WinRAR/7-Zip for Windows Zipeg/iZip/UnRarX for Mac 7-Zip/PeaZip for Linux The code bundle for the book is also hosted on GitHub at https:/​/​github.​com/ PacktPublishing/​Learning-​Path-​Learn-​Web-​Development-​with-​Python. In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https:/​/​github.​com/​PacktPublishing/​. Check them out!

Conventions used There are a number of text conventions used throughout this book. CodeInText: Indicates code words in text, database table names, folder names,

filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Some common annotations are @Service, @Component, @Bean, and @Configuration." A block of code is set as follows: http .formLogin() .loginPage("/login") .failureUrl("/login?error") .and() .authorizeRequests() .antMatchers("/signup","/about").permitAll() .antMatchers("/admin/**").hasRole("ADMIN") .anyRequest().authenticated();

[7]

Preface

Any command-line input or output is written as follows: sudo apt-get install openjdk-8-jdk -y java -version

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "In the Project Metadata section, we can put the coordinates for Maven projects." Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch Feedback from our readers is always welcome. General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected]. Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details. Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material. If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

[8]

Preface

Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about Packt, please visit packt.com.

[9]

1 A Gentle Introduction to Python "Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime." – Chinese proverb According to Wikipedia, computer programming is: "...a process that leads from an original formulation of a computing problem to executable computer programs. Programming involves activities such as analysis, developing understanding, generating algorithms, verification of requirements of algorithms including their correctness and resources consumption, and implementation (commonly referred to as coding) of algorithms in a target programming language." In a nutshell, coding is telling a computer to do something using a language it understands. Computers are very powerful tools, but unfortunately, they can't think for themselves. They need to be told everything: how to perform a task, how to evaluate a condition to decide which path to follow, how to handle data that comes from a device, such as the network or a disk, and how to react when something unforeseen happens, say, something is broken or missing. You can code in many different styles and languages. Is it hard? I would say yes and no. It's a bit like writing. Everybody can learn how to write, and you can too. But, what if you wanted to become a poet? Then writing alone is not enough. You have to acquire a whole other set of skills and this will take a longer and greater effort.

A Gentle Introduction to Python

Chapter 1

In the end, it all comes down to how far you want to go down the road. Coding is not just putting together some instructions that work. It is so much more! Good code is short, fast, elegant, easy to read and understand, simple, easy to modify and extend, easy to scale and refactor, and easy to test. It takes time to be able to write code that has all these qualities at the same time, but the good news is that you're taking the first step towards it at this very moment by reading this book. And I have no doubt you can do it. Anyone can; in fact, we all program all the time, only we aren't aware of it. Would you like an example? Say you want to make instant coffee. You have to get a mug, the instant coffee jar, a teaspoon, water, and the kettle. Even if you're not aware of it, you're evaluating a lot of data. You're making sure that there is water in the kettle and that the kettle is plugged in, that the mug is clean, and that there is enough coffee in the jar. Then, you boil the water and maybe, in the meantime, you put some coffee in the mug. When the water is ready, you pour it into the cup, and stir. So, how is this programming? Well, we gathered resources (the kettle, coffee, water, teaspoon, and mug) and we verified some conditions concerning them (the kettle is plugged in, the mug is clean, and there is enough coffee). Then we started two actions (boiling the water and putting coffee in the mug), and when both of them were completed, we finally ended the procedure by pouring water in to the mug and stirring. Can you see it? I have just described the high-level functionality of a coffee program. It wasn't that hard because this is what the brain does all day long: evaluate conditions, decide to take actions, carry out tasks, repeat some of them, and stop at some point. Clean objects, put them back, and so on. All you need now is to learn how to deconstruct all those actions you do automatically in real life so that a computer can actually make some sense of them. And you need to learn a language as well, to instruct it. So this is what this book is for. I'll tell you how to do it and I'll try to do that by means of many simple but focused examples (my favorite kind). In this chapter, we are going to cover the following: Python's characteristics and ecosystem Guidelines on how to get up and running with Python and virtual environments

[ 11 ]

A Gentle Introduction to Python

Chapter 1

How to run Python programs How to organize Python code and Python's execution model

A proper introduction I love to make references to the real world when I teach coding; I believe they help people retain the concepts better. However, now is the time to be a bit more rigorous and see what coding is from a more technical perspective. When we write code, we're instructing a computer about the things it has to do. Where does the action happen? In many places: the computer memory, hard drives, network cables, the CPU, and so on. It's a whole world, which most of the time is the representation of a subset of the real world. If you write a piece of software that allows people to buy clothes online, you will have to represent real people, real clothes, real brands, sizes, and so on and so forth, within the boundaries of a program. In order to do so, you will need to create and handle objects in the program you're writing. A person can be an object. A car is an object. A pair of socks is an object. Luckily, Python understands objects very well. The two main features any object has are properties and methods. Let's take a person object as an example. Typically in a computer program, you'll represent people as customers or employees. The properties that you store against them are things like the name, the SSN, the age, if they have a driving license, their email, gender, and so on. In a computer program, you store all the data you need in order to use an object for the purpose you're serving. If you are coding a website to sell clothes, you probably want to store the heights and weights as well as other measures of your customers so that you can suggest the appropriate clothes for them. So, properties are characteristics of an object. We use them all the time: Could you pass me that pen?—Which one?—The black one. Here, we used the black property of a pen to identify it (most likely among a blue and a red one). Methods are things that an object can do. As a person, I have methods such as speak, walk, sleep, wake up, eat, dream, write, read, and so on. All the things that I can do could be seen as methods of the objects that represent me.

[ 12 ]

A Gentle Introduction to Python

Chapter 1

So, now that you know what objects are and that they expose methods that you can run and properties that you can inspect, you're ready to start coding. Coding in fact is simply about managing those objects that live in the subset of the world that we're reproducing in our software. You can create, use, reuse, and delete objects as you please. According to the Data Model chapter on the official Python documentation (https:/​/ docs.​python.​org/​3/​reference/​datamodel.​html): "Objects are Python's abstraction for data. All data in a Python program is represented by objects or by relations between objects." We'll take a closer look at Python objects in Chapter 6, OOP, Decorators, and Iterators. For now, all we need to know is that every object in Python has an ID (or identity), a type, and a value. Once created, the ID of an object is never changed. It's a unique identifier for it, and it's used behind the scenes by Python to retrieve the object when we want to use it. The type, as well, never changes. The type tells what operations are supported by the object and the possible values that can be assigned to it. We'll see Python's most important data types in Chapter 2, Built-in Data Types. The value can either change or not. If it can, the object is said to be mutable, while when it cannot, the object is said to be immutable. How do we use an object? We give it a name, of course! When you give an object a name, then you can use the name to retrieve the object and use it. In a more generic sense, objects such as numbers, strings (text), collections, and so on are associated with a name. Usually, we say that this name is the name of a variable. You can see the variable as being like a box, which you can use to hold data. So, you have all the objects you need; what now? Well, we need to use them, right? We may want to send them over a network connection or store them in a database. Maybe display them on a web page or write them into a file. In order to do so, we need to react to a user filling in a form, or pressing a button, or opening a web page and performing a search. We react by running our code, evaluating conditions to choose which parts to execute, how many times, and under which circumstances. And to do all this, basically we need a language. That's what Python is for. Python is the language we'll use together throughout this book to instruct the computer to do something for us.

[ 13 ]

A Gentle Introduction to Python

Chapter 1

Now, enough of this theoretical stuff; let's get started.

Enter the Python Python is the marvelous creation of Guido Van Rossum, a Dutch computer scientist and mathematician who decided to gift the world with a project he was playing around with over Christmas 1989. The language appeared to the public somewhere around 1991, and since then has evolved to be one of the leading programming languages used worldwide today. I started programming when I was 7 years old, on a Commodore VIC-20, which was later replaced by its bigger brother, the Commodore 64. Its language was BASIC. Later on, I landed on Pascal, Assembly, C, C++, Java, JavaScript, Visual Basic, PHP, ASP, ASP .NET, C#, and other minor languages I cannot even remember, but only when I landed on Python did I finally have that feeling that you have when you find the right couch in the shop. When all of your body parts are yelling, Buy this one! This one is perfect for us! It took me about a day to get used to it. Its syntax is a bit different from what I was used to, but after getting past that initial feeling of discomfort (like having new shoes), I just fell in love with it. Deeply. Let's see why.

About Python Before we get into the gory details, let's get a sense of why someone would want to use Python (I would recommend you to read the Python page on Wikipedia to get a more detailed introduction). To my mind, Python epitomizes the following qualities.

Portability Python runs everywhere, and porting a program from Linux to Windows or Mac is usually just a matter of fixing paths and settings. Python is designed for portability and it takes care of specific operating system (OS) quirks behind interfaces that shield you from the pain of having to write code tailored to a specific platform.

[ 14 ]

A Gentle Introduction to Python

Chapter 1

Coherence Python is extremely logical and coherent. You can see it was designed by a brilliant computer scientist. Most of the time, you can just guess how a method is called, if you don't know it. You may not realize how important this is right now, especially if you are at the beginning, but this is a major feature. It means less cluttering in your head, as well as less skimming through the documentation, and less need for mappings in your brain when you code.

Developer productivity According to Mark Lutz (Learning Python, 5th Edition, O'Reilly Media), a Python program is typically one-fifth to one-third the size of equivalent Java or C++ code. This means the job gets done faster. And faster is good. Faster means a faster response on the market. Less code not only means less code to write, but also less code to read (and professional coders read much more than they write), less code to maintain, to debug, and to refactor. Another important aspect is that Python runs without the need for lengthy and timeconsuming compilation and linkage steps, so you don't have to wait to see the results of your work.

An extensive library Python has an incredibly wide standard library (it's said to come with batteries included). If that wasn't enough, the Python community all over the world maintains a body of third-party libraries, tailored to specific needs, which you can access freely at the Python Package Index (PyPI). When you code Python and you realize that you need a certain feature, in most cases, there is at least one library where that feature has already been implemented for you.

[ 15 ]

A Gentle Introduction to Python

Chapter 1

Software quality Python is heavily focused on readability, coherence, and quality. The language uniformity allows for high readability and this is crucial nowadays where coding is more of a collective effort than a solo endeavor. Another important aspect of Python is its intrinsic multiparadigm nature. You can use it as a scripting language, but you also can exploit object-oriented, imperative, and functional programming styles. It is versatile.

Software integration Another important aspect is that Python can be extended and integrated with many other languages, which means that even when a company is using a different language as their mainstream tool, Python can come in and act as a glue agent between complex applications that need to talk to each other in some way. This is kind of an advanced topic, but in the real world, this feature is very important.

Satisfaction and enjoyment Last, but not least, there is the fun of it! Working with Python is fun. I can code for 8 hours and leave the office happy and satisfied, alien to the struggle other coders have to endure because they use languages that don't provide them with the same amount of well-designed data structures and constructs. Python makes coding fun, no doubt about it. And fun promotes motivation and productivity. These are the major aspects of why I would recommend Python to everyone. Of course, there are many other technical and advanced features that I could have talked about, but they don't really pertain to an introductory section like this one. They will come up naturally, chapter after chapter, in this book.

What are the drawbacks? Probably, the only drawback that one could find in Python, which is not due to personal preferences, is its execution speed. Typically, Python is slower than its compiled brothers. The standard implementation of Python produces, when you run an application, a compiled version of the source code called byte code (with the extension .pyc), which is then run by the Python interpreter.

[ 16 ]

A Gentle Introduction to Python

Chapter 1

The advantage of this approach is portability, which we pay for with a slowdown due to the fact that Python is not compiled down to machine level as are other languages. However, Python speed is rarely a problem today, hence its wide use regardless of this suboptimal feature. What happens is that, in real life, hardware cost is no longer a problem, and usually it's easy enough to gain speed by parallelizing tasks. Moreover, many programs spend a great proportion of the time waiting for IO operations to complete; therefore, the raw execution speed is often a secondary factor to the overall performance. When it comes to number crunching though, one can switch to faster Python implementations, such as PyPy, which provides an average five-fold speedup by implementing advanced compilation techniques (check http://pypy.org/ for reference). When doing data science, you'll most likely find that the libraries that you use with Python, such as Pandas and NumPy, achieve native speed due to the way they are implemented. If that wasn't a good-enough argument, you can always consider that Python has been used to drive the backend of services such as Spotify and Instagram, where performance is a concern. Nonetheless, Python has done its job perfectly adequately.

Who is using Python today? Not yet convinced? Let's take a very brief look at the companies that are using Python today: Google, YouTube, Dropbox, Yahoo!, Zope Corporation, Industrial Light & Magic, Walt Disney Feature Animation, Blender 3D, Pixar, NASA, the NSA, Red Hat, Nokia, IBM, Netflix, Yelp, Intel, Cisco, HP, Qualcomm, and JPMorgan Chase, to name just a few. Even games such as Battlefield 2, Civilization IV, and QuArK are implemented using Python. Python is used in many different contexts, such as system programming, web programming, GUI applications, gaming and robotics, rapid prototyping, system integration, data science, database applications, and much more. Several prestigious universities have also adopted Python as their main language in computer science courses.

[ 17 ]

A Gentle Introduction to Python

Chapter 1

Setting up the environment Before we talk about installing Python on your system, let me tell you about which Python version I'll be using in this book.

Python 2 versus Python 3 Python comes in two main versions: Python 2, which is the past, and Python 3, which is the present. The two versions, though very similar, are incompatible in some respects. In the real world, Python 2 is actually quite far from being the past. In short, even though Python 3 has been out since 2008, the transition phase from Version 2 is still far from being over. This is mostly due to the fact that Python 2 is widely used in the industry, and of course, companies aren't so keen on updating their systems just for the sake of updating them, following the if it ain't broke, don't fix it philosophy. You can read all about the transition between the two versions on the web. Another issue that has hindered the transition is the availability of third-party libraries. Usually, a Python project relies on tens of external libraries, and of course, when you start a new project, you need to be sure that there is already a Version-3compatible library for any business requirement that may come up. If that's not the case, starting a brand-new project in Python 3 means introducing a potential risk, which many companies are not happy to take. At the time of writing, though, the majority of the most widely used libraries have been ported to Python 3, and it's quite safe to start a project in Python 3 for most cases. Many of the libraries have been rewritten so that they are compatible with both versions, mostly harnessing the power of the six library (the name comes from the multiplication 2 x 3, due to the porting from Version 2 to 3), which helps introspecting and adapting the behavior according to the version used. According to PEP 373 (https:/​/​legacy.​python.​org/​dev/​peps/​pep-​0373/​), the end of life (EOL) of Python 2.7 has been set to 2020, and there won't be a Python 2.8, so this is the time when companies that have projects running in Python 2 need to start devising an upgrade strategy to move to Python 3 before it's too late.

[ 18 ]

A Gentle Introduction to Python

Chapter 1

On my box (MacBook Pro), this is the latest Python version I have: >>> import sys >>> print(sys.version) 3.7.0a3 (default, Jan 27 2018, 00:46:45) [Clang 9.0.0 (clang-900.0.39.2)]

So you can see that the version is an alpha release of Python 3.7, which will be released in June 2018. The preceding text is a little bit of Python code that I typed into my console. We'll talk about it in a moment. All the examples in this book will be run using Python 3.7. Even though at the moment the final version might still be slightly different than what I have, I will make sure that all the code and examples are up to date with 3.7 by the time the book is published. Some of the code can also run in Python 2.7, either as it is or with minor tweaks, but at this point in time, I think it's better to learn Python 3, and then, if you need to, learn the differences it has with Python 2, rather than going the other way around. Don't worry about this version thing though; it's not that big an issue in practice.

Installing Python I never really got the point of having a setup section in a book, regardless of what it is that you have to set up. Most of the time, between the time the author writes the instructions and the time you actually try them out, months have passed. That is, if you're lucky. One version change and things may not work in the way that is described in the book. Luckily, we have the web now, so in order to help you get up and running, I'll just give you pointers and objectives. I am conscious that the majority of readers would probably have preferred to have guidelines in the book. I doubt it would have made their life much easier, as I strongly believe that if you want to get started with Python you have to put in that initial effort in order to get familiar with the ecosystem. It is very important, and it will boost your confidence to face the material in the chapters ahead. If you get stuck, remember that Google is your friend.

[ 19 ]

A Gentle Introduction to Python

Chapter 1

Setting up the Python interpreter First of all, let's talk about your OS. Python is fully integrated and most likely already installed in basically almost every Linux distribution. If you have a macOS, it's likely that Python is already there as well (however, possibly only Python 2.7), whereas if you're using Windows, you probably need to install it. Getting Python and the libraries you need up and running requires a bit of handiwork. Linux and macOS seem to be the most user-friendly OSes for Python programmers; Windows, on the other hand, is the one that requires the biggest effort. My current system is a MacBook Pro, and this is what I will use throughout the book, along with Python 3.7. The place you want to start is the official Python website: https://www.python.org. This website hosts the official Python documentation and many other resources that you will find very useful. Take the time to explore it. Another excellent, resourceful website on Python and its ecosystem is http://docs.python-guide.org. You can find instructions to set up Python on different operating systems, using different methods. Find the download section and choose the installer for your OS. If you are on Windows, make sure that when you run the installer, you check the option install pip (actually, I would suggest to make a complete installation, just to be safe, of all the components the installer holds). We'll talk about pip later. Now that Python is installed in your system, the objective is to be able to open a console and run the Python interactive shell by typing python. Please note that I usually refer to the Python interactive shell simply as the Python console.

[ 20 ]

A Gentle Introduction to Python

Chapter 1

To open the console in Windows, go to the Start menu, choose Run, and type cmd. If you encounter anything that looks like a permission problem while working on the examples in this book, please make sure you are running the console with administrator rights. On the macOS X, you can start a Terminal by going to Applications | Utilities | Terminal. If you are on Linux, you know all that there is to know about the console. I will use the term console interchangeably to indicate the Linux console, the Windows Command Prompt, and the Macintosh Terminal. I will also indicate the commandline prompt with the Linux default format, like this: $ sudo apt-get update

If you're not familiar with that, please take some time to learn the basics on how a console works. In a nutshell, after the $ sign, you normally find an instruction that you have to type. Pay attention to capitalization and spaces, as they are very important. Whatever console you open, type python at the prompt, and make sure the Python interactive shell shows up. Type exit() to quit. Keep in mind that you may have to specify python3 if your OS comes with Python 2.* preinstalled. This is roughly what you should see when you run Python (it will change in some details according to the version and OS): $ python3.7 Python 3.7.0a3 (default, Jan 27 2018, 00:46:45) [Clang 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

[ 21 ]

A Gentle Introduction to Python

Chapter 1

Now that Python is set up and you can run it, it's time to make sure you have the other tool that will be indispensable to follow the examples in the book: virtualenv.

About virtualenv As you probably have guessed by its name, virtualenv is all about virtual environments. Let me explain what they are and why we need them and let me do it by means of a simple example. You install Python on your system and you start working on a website for Client X. You create a project folder and start coding. Along the way, you also install some libraries; for example, the Django framework, which we'll see in depth in Chapter 14, Web Development. Let's say the Django version you install for Project X is 1.7.1. Now, your website is so good that you get another client, Y. She wants you to build another website, so you start Project Y and, along the way, you need to install Django again. The only issue is that now the Django version is 1.8 and you cannot install it on your system because this would replace the version you installed for Project X. You don't want to risk introducing incompatibility issues, so you have two choices: either you stick with the version you have currently on your machine, or you upgrade it and make sure the first project is still fully working correctly with the new version. Let's be honest, neither of these options is very appealing, right? Definitely not. So, here's the solution: virtualenv! virtualenv is a tool that allows you to create a virtual environment. In other words, it is a tool to create isolated Python environments, each of which is a folder that contains all the necessary executables to use the packages that a Python project would need (think of packages as libraries for the time being). So you create a virtual environment for Project X, install all the dependencies, and then you create a virtual environment for Project Y, installing all its dependencies without the slightest worry because every library you install ends up within the boundaries of the appropriate virtual environment. In our example, Project X will hold Django 1.7.1, while Project Y will hold Django 1.8.

[ 22 ]

A Gentle Introduction to Python

Chapter 1

It is of vital importance that you never install libraries directly at the system level. Linux, for example, relies on Python for many different tasks and operations, and if you fiddle with the system installation of Python, you risk compromising the integrity of the whole system (guess to whom this happened...). So take this as a rule, such as brushing your teeth before going to bed: always, always create a virtual environment when you start a new project. To install virtualenv on your system, there are a few different ways. On a Debianbased distribution of Linux, for example, you can install it with the following command: $ sudo apt-get install python-virtualenv

Probably, the easiest way is to follow the instructions you can find on the virtualenv official website: https:/​/​virtualenv.​pypa.​io. You will find that one of the most common ways to install virtualenv is by using pip, a package management system used to install and manage software packages written in Python. As of Python 3.5, the suggested way to create a virtual environment is to use the venv module. Please see the official documentation for further information. However, at the time of writing, virtualenv is still by far the tool most used for creating virtual environments.

Your first virtual environment It is very easy to create a virtual environment, but according to how your system is configured and which Python version you want the virtual environment to run, you need to run the command properly. Another thing you will need to do with virtualenv, when you want to work with it, is to activate it. Activating virtualenv basically produces some path juggling behind the scenes so that when you call the Python interpreter, you're actually calling the active virtual environment one, instead of the mere system one.

[ 23 ]

A Gentle Introduction to Python

Chapter 1

I'll show you a full example on my Macintosh console. We will: 1. Create a folder named learn.pp under your project root (which in my case is a folder called srv, in my home folder). Please adapt the paths according to the setup you fancy on your box. 2. Within the learn.pp folder, we will create a virtual environment called learnpp. Some developers prefer to call all virtual environments using the same name (for example, .venv). This way they can run scripts against any virtualenv by just knowing the name of the project they dwell in. The dot in .venv is there because in Linux/macOS prepending a name with a dot makes that file or folder invisible. 3. After creating the virtual environment, we will activate it. The methods are slightly different between Linux, macOS, and Windows. 4. Then, we'll make sure that we are running the desired Python version (3.7.*) by running the Python interactive shell. 5. Finally, we will deactivate the virtual environment using the deactivate command. These five simple steps will show you all you have to do to start and use a project. Here's an example of how those steps might look (note that you might get a slightly different result, according to your OS, Python version, and so on) on the macOS (commands that start with a # are comments, spaces have been introduced for readability, and ⇢ indicates where the line has wrapped around due to lack of space): fabmp:srv fab$ # step 1 - create folder fabmp:srv fab$ mkdir learn.pp fabmp:srv fab$ cd learn.pp fabmp:learn.pp fab$ # step 2 - create virtual environment fabmp:learn.pp fab$ which python3.7 /Users/fab/.pyenv/shims/python3.7 fabmp:learn.pp fab$ virtualenv -p ⇢ /Users/fab/.pyenv/shims/python3.7 learnpp Running virtualenv with interpreter /Users/fab/.pyenv/shims/python3.7 Using base prefix '/Users/fab/.pyenv/versions/3.7.0a3' New python executable in /Users/fab/srv/learn.pp/learnpp/bin/python3.7 Also creating executable in /Users/fab/srv/learn.pp/learnpp/bin/python Installing setuptools, pip, wheel...done. fabmp:learn.pp fab$ # step 3 - activate virtual environment

[ 24 ]

A Gentle Introduction to Python

Chapter 1

fabmp:learn.pp fab$ source learnpp/bin/activate (learnpp) fabmp:learn.pp fab$ # step 4 - verify which python (learnpp) fabmp:learn.pp fab$ which python /Users/fab/srv/learn.pp/learnpp/bin/python (learnpp) fabmp:learn.pp fab$ python Python 3.7.0a3 (default, Jan 27 2018, 00:46:45) [Clang 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> exit() (learnpp) fabmp:learn.pp fab$ # step 5 - deactivate (learnpp) fabmp:learn.pp fab$ deactivate fabmp:learn.pp fab$

Notice that I had to tell virtualenv explicitly to use the Python 3.7 interpreter because on my box Python 2.7 is the default one. Had I not done that, I would have had a virtual environment with Python 2.7 instead of Python 3.7. You can combine the two instructions for step 2 in one single command like this: $ virtualenv -p $( which python3.7 ) learnpp

I chose to be explicitly verbose in this instance, to help you understand each bit of the procedure. Another thing to notice is that in order to activate a virtual environment, we need to run the /bin/activate script, which needs to be sourced. When a script is sourced, it means that it is executed in the current shell, and therefore its effects last after the execution. This is very important. Also notice how the prompt changes after we activate the virtual environment, showing its name on the left (and how it disappears when we deactivate it). On Linux, the steps are the same so I won't repeat them here. On Windows, things change slightly, but the concepts are the same. Please refer to the official virtualenv website for guidance. At this point, you should be able to create and activate a virtual environment. Please try and create another one without me guiding you. Get acquainted with this procedure because it's something that you will always be doing: we never work system-wide with Python, remember? It's extremely important. So, with the scaffolding out of the way, we're ready to talk a bit more about Python and how you can use it. Before we do that though, allow me to speak a few words about the console.

[ 25 ]

A Gentle Introduction to Python

Chapter 1

Your friend, the console In this era of GUIs and touchscreen devices, it seems a little ridiculous to have to resort to a tool such as the console, when everything is just about one click away. But the truth is every time you remove your right hand from the keyboard (or the left one, if you're a lefty) to grab your mouse and move the cursor over to the spot you want to click on, you're losing time. Getting things done with the console, counterintuitive as it may be, results in higher productivity and speed. I know, you have to trust me on this. Speed and productivity are important and, personally, I have nothing against the mouse, but there is another very good reason for which you may want to get wellacquainted with the console: when you develop code that ends up on some server, the console might be the only available tool. If you make friends with it, I promise you, you will never get lost when it's of utmost importance that you don't (typically, when the website is down and you have to investigate very quickly what's going on). So it's really up to you. If you're undecided, please grant me the benefit of the doubt and give it a try. It's easier than you think, and you'll never regret it. There is nothing more pitiful than a good developer who gets lost within an SSH connection to a server because they are used to their own custom set of tools, and only to that. Now, let's get back to Python.

How you can run a Python program There are a few different ways in which you can run a Python program.

Running Python scripts Python can be used as a scripting language. In fact, it always proves itself very useful. Scripts are files (usually of small dimensions) that you normally execute to do something like a task. Many developers end up having their own arsenal of tools that they fire when they need to perform a task. For example, you can have scripts to parse data in a format and render it into another different format. Or you can use a script to work with files and folders. You can create or modify configuration files, and much more. Technically, there is not much that cannot be done in a script.

[ 26 ]

A Gentle Introduction to Python

Chapter 1

It's quite common to have scripts running at a precise time on a server. For example, if your website database needs cleaning every 24 hours (for example, the table that stores the user sessions, which expire pretty quickly but aren't cleaned automatically), you could set up a Cron job that fires your script at 3:00 A.M. every day. According to Wikipedia, the software utility Cron is a time-based job scheduler in Unix-like computer operating systems. People who set up and maintain software environments use Cron to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals. I have Python scripts to do all the menial tasks that would take me minutes or more to do manually, and at some point, I decided to automate. We'll devote half of Chapter 12, GUIs and Scripts, on scripting with Python.

Running the Python interactive shell Another way of running Python is by calling the interactive shell. This is something we already saw when we typed python on the command line of our console. So, open a console, activate your virtual environment (which by now should be second nature to you, right?), and type python. You will be presented with a couple of lines that should look like this: $ python Python 3.7.0a3 (default, Jan 27 2018, 00:46:45) [Clang 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

Those >>> are the prompt of the shell. They tell you that Python is waiting for you to type something. If you type a simple instruction, something that fits in one line, that's all you'll see. However, if you type something that requires more than one line of code, the shell will change the prompt to ..., giving you a visual clue that you're typing a multiline statement (or anything that would require more than one line of code). Go on, try it out; let's do some basic math: >>> 2 + 4 6 >>> 10 / 4 2.5

[ 27 ]

A Gentle Introduction to Python

Chapter 1

>>> 2 ** 1024 1797693134862315907729305190789024733617976978942306572734300811577326 7580550096313270847732240753602112011387987139335765878976881441662249 2847430639474124377767893424865485276302219601246094119453082952085005 7688381506823424628814739131105408272371633505106845862982399472459384 79716304835356329624224137216

The last operation is showing you something incredible. We raise 2 to the power of 1024, and Python is handling this task with no trouble at all. Try to do it in Java, C++, or C#. It won't work, unless you use special libraries to handle such big numbers. I use the interactive shell every day. It's extremely useful to debug very quickly, for example, to check if a data structure supports an operation. Or maybe to inspect or run a piece of code. When you use Django (a web framework), the interactive shell is coupled with it and allows you to work your way through the framework tools, to inspect the data in the database, and many more things. You will find that the interactive shell will soon become one of your dearest friends on the journey you are embarking on. Another solution, which comes in a much nicer graphic layout, is to use Integrated DeveLopment Environment (IDLE). It's quite a simple IDE, which is intended mostly for beginners. It has a slightly larger set of capabilities than the naked interactive shell you get in the console, so you may want to explore it. It comes for free in the Windows Python installer and you can easily install it in any other system. You can find information about it on the Python website. Guido Van Rossum named Python after the British comedy group, Monty Python, so it's rumored that the name IDLE has been chosen in honor of Eric Idle, one of Monty Python's founding members.

Running Python as a service Apart from being run as a script, and within the boundaries of a shell, Python can be coded and run as an application. We'll see many examples throughout the book about this mode. And we'll understand more about it in a moment, when we'll talk about how Python code is organized and run.

[ 28 ]

A Gentle Introduction to Python

Chapter 1

Running Python as a GUI application Python can also be run as a graphical user interface (GUI). There are several frameworks available, some of which are cross-platform and some others are platform-specific. In Chapter 12, GUIs and Scripts, we'll see an example of a GUI application created using Tkinter, which is an object-oriented layer that lives on top of Tk (Tkinter means Tk interface). Tk is a GUI toolkit that takes desktop application development to a higher level than the conventional approach. It is the standard GUI for Tool Command Language (Tcl), but also for many other dynamic languages, and it can produce rich native applications that run seamlessly under Windows, Linux, macOS X, and more. Tkinter comes bundled with Python; therefore, it gives the programmer easy access to the GUI world, and for these reasons, I have chosen it to be the framework for the GUI examples that I'll present in this book. Among the other GUI frameworks, we find that the following are the most widely used: PyQt wxPython PyGTK Describing them in detail is outside the scope of this book, but you can find all the information you need on the Python website (https:/​/​docs.​python.​org/​3/​faq/​gui. html) in the What platform-independent GUI toolkits exist for Python? section. If GUIs are what you're looking for, remember to choose the one you want according to some principles. Make sure they: Offer all the features you may need to develop your project Run on all the platforms you may need to support Rely on a community that is as wide and active as possible Wrap graphic drivers/tools that you can easily install/access

How is Python code organized? Let's talk a little bit about how Python code is organized. In this section, we'll start going down the rabbit hole a little bit more and introduce more technical names and concepts.

[ 29 ]

A Gentle Introduction to Python

Chapter 1

Starting with the basics, how is Python code organized? Of course, you write your code into files. When you save a file with the extension .py, that file is said to be a Python module. If you're on Windows or macOS that typically hide file extensions from the user, please make sure you change the configuration so that you can see the complete names of the files. This is not strictly a requirement, but a suggestion. It would be impractical to save all the code that it is required for software to work within one single file. That solution works for scripts, which are usually not longer than a few hundred lines (and often they are quite shorter than that). A complete Python application can be made of hundreds of thousands of lines of code, so you will have to scatter it through different modules, which is better, but not nearly good enough. It turns out that even like this, it would still be impractical to work with the code. So Python gives you another structure, called package, which allows you to group modules together. A package is nothing more than a folder, which must contain a special file, __init__.py, that doesn't need to hold any code but whose presence is required to tell Python that the folder is not just some folder, but it's actually a package (note that as of Python 3.3, the __init__.py module is not strictly required any more). As always, an example will make all of this much clearer. I have created an example structure in my book project, and when I type in my console: $ tree -v example

I get a tree representation of the contents of the ch1/example folder, which holds the code for the examples of this chapter. Here's what the structure of a really simple application could look like: example ├── core.py ├── run.py └── util ├── __init__.py ├── db.py ├── math.py └── network.py

[ 30 ]

A Gentle Introduction to Python

Chapter 1

You can see that within the root of this example, we have two modules, core.py and run.py, and one package: util. Within core.py, there may be the core logic of our application. On the other hand, within the run.py module, we can probably find the logic to start the application. Within the util package, I expect to find various utility tools, and in fact, we can guess that the modules there are named based on the types of tools they hold: db.py would hold tools to work with databases, math.py would, of course, hold mathematical tools (maybe our application deals with financial data), and network.py would probably hold tools to send/receive data on networks. As explained before, the __init__.py file is there just to tell Python that util is a package and not just a mere folder. Had this software been organized within modules only, it would have been harder to infer its structure. I put a module only example under the ch1/files_only folder; see it for yourself: $ tree -v files_only

This shows us a completely different picture: files_only/ ├── core.py ├── db.py ├── math.py ├── network.py └── run.py

It is a little harder to guess what each module does, right? Now, consider that this is just a simple example, so you can guess how much harder it would be to understand a real application if we couldn't organize the code in packages and modules.

How do we use modules and packages? When a developer is writing an application, it is likely that they will need to apply the same piece of logic in different parts of it. For example, when writing a parser for the data that comes from a form that a user can fill in a web page, the application will have to validate whether a certain field is holding a number or not. Regardless of how the logic for this kind of validation is written, it's likely that it will be needed in more than one place.

[ 31 ]

A Gentle Introduction to Python

Chapter 1

For example, in a poll application, where the user is asked many questions, it's likely that several of them will require a numeric answer. For example: What is your age? How many pets do you own? How many children do you have? How many times have you been married? It would be very bad practice to copy/paste (or, more properly said: duplicate) the validation logic in every place where we expect a numeric answer. This would violate the don't repeat yourself (DRY) principle, which states that you should never repeat the same piece of code more than once in your application. I feel the need to stress the importance of this principle: you should never repeat the same piece of code more than once in your application (pun intended). There are several reasons why repeating the same piece of logic can be very bad, the most important ones being: There could be a bug in the logic, and therefore, you would have to correct it in every place that the logic is applied. You may want to amend the way you carry out the validation, and again you would have to change it in every place it is applied. You may forget to fix/amend a piece of logic because you missed it when searching for all its occurrences. This would leave wrong/inconsistent behavior in your application. Your code would be longer than needed, for no good reason. Python is a wonderful language and provides you with all the tools you need to apply all the coding best practices. For this particular example, we need to be able to reuse a piece of code. To be able to reuse a piece of code, we need to have a construct that will hold the code for us so that we can call that construct every time we need to repeat the logic inside it. That construct exists, and it's called a function. I'm not going too deep into the specifics here, so please just remember that a function is a block of organized, reusable code that is used to perform a task. Functions can assume many forms and names, according to what kind of environment they belong to, but for now this is not important. We'll see the details when we are able to appreciate them, later on, in the book. Functions are the building blocks of modularity in your application, and they are almost indispensable. Unless you're writing a supersimple script, you'll use functions all the time. We'll explore functions in Chapter 4, Functions, the Building Blocks of Code.

[ 32 ]

A Gentle Introduction to Python

Chapter 1

Python comes with a very extensive library, as I have already said a few pages ago. Now, maybe it's a good time to define what a library is: a library is a collection of functions and objects that provide functionalities that enrich the abilities of a language. For example, within Python's math library, we can find a plethora of functions, one of which is the factorial function, which of course calculates the factorial of a number. In mathematics, the factorial of a non-negative integer number N, denoted as N!, is defined as the product of all positive integers less than or equal to N. For example, the factorial of 5 is calculated as: 5! = 5 * 4 * 3 * 2 * 1 = 120 The factorial of 0 is 0! = 1, to respect the convention for an empty product. So, if you wanted to use this function in your code, all you would have to do is to import it and call it with the right input values. Don't worry too much if input values and the concept of calling is not very clear for now; please just concentrate on the import part. We use a library by importing what we need from it, and then we use it. In Python, to calculate the factorial of number 5, we just need the following code: >>> from math import factorial >>> factorial(5) 120

Whatever we type in the shell, if it has a printable representation, will be printed on the console for us (in this case, the result of the function call: 120). So, let's go back to our example, the one with core.py, run.py, util, and so on. In our example, the package util is our utility library. Our custom utility belt that holds all those reusable tools (that is, functions), which we need in our application. Some of them will deal with databases (db.py), some with the network (network.py), and some will perform mathematical calculations (math.py) that are outside the scope of Python's standard math library and, therefore, we have to code them for ourselves. We will see in detail how to import functions and use them in their dedicated chapter. Let's now talk about another very important concept: Python's execution model.

[ 33 ]

A Gentle Introduction to Python

Chapter 1

Python's execution model In this section, I would like to introduce you to a few very important concepts, such as scope, names, and namespaces. You can read all about Python's execution model in the official language reference, of course, but I would argue that it is quite technical and abstract, so let me give you a less formal explanation first.

Names and namespaces Say you are looking for a book, so you go to the library and ask someone for the book you want to fetch. They tell you something like Second Floor, Section X, Row Three. So you go up the stairs, look for Section X, and so on. It would be very different to enter a library where all the books are piled together in random order in one big room. No floors, no sections, no rows, no order. Fetching a book would be extremely hard. When we write code, we have the same issue: we have to try and organize it so that it will be easy for someone who has no prior knowledge about it to find what they're looking for. When software is structured correctly, it also promotes code reuse. On the other hand, disorganized software is more likely to expose scattered pieces of duplicated logic. First of all, let's start with the book. We refer to a book by its title and in Python lingo, that would be a name. Python names are the closest abstraction to what other languages call variables. Names basically refer to objects and are introduced by namebinding operations. Let's make a quick example (notice that anything that follows a # is a comment): >>> n = 3 # integer number >>> address = "221b Baker Street, NW1 6XE, London" address >>> employee = { ... 'age': 45, ... 'role': 'CTO', ... 'SSN': 'AB1234567', ... } >>> # let's print them >>> n 3 >>> address '221b Baker Street, NW1 6XE, London' >>> employee

[ 34 ]

# Sherlock Holmes'

A Gentle Introduction to Python

Chapter 1

{'age': 45, 'role': 'CTO', 'SSN': 'AB1234567'} >>> other_name Traceback (most recent call last): File "", line 1, in NameError: name 'other_name' is not defined

We defined three objects in the preceding code (do you remember what are the three features every Python object has?): An integer number n (type: int, value: 3) A string address (type: str, value: Sherlock Holmes' address) A dictionary employee (type: dict, value: a dictionary that holds three key/value pairs) Don't worry, I know you're not supposed to know what a dictionary is. We'll see in Chapter 2, Built-in Data Types, that it's the king of Python data structures. Have you noticed that the prompt changed from >>> to ... when I typed in the definition of employee? That's because the definition spans over multiple lines. So, what are n, address, and employee? They are names. Names that we can use to retrieve data within our code. They need to be kept somewhere so that whenever we need to retrieve those objects, we can use their names to fetch them. We need some space to hold them, hence: namespaces! A namespace is therefore a mapping from names to objects. Examples are the set of built-in names (containing functions that are always accessible in any Python program), the global names in a module, and the local names in a function. Even the set of attributes of an object can be considered a namespace. The beauty of namespaces is that they allow you to define and organize your names with clarity, without overlapping or interference. For example, the namespace associated with that book we were looking for in the library can be used to import the book itself, like this: from library.second_floor.section_x.row_three import book

We start from the library namespace, and by means of the dot (.) operator, we walk into that namespace. Within this namespace, we look for second_floor, and again we walk into it with the . operator. We then walk into section_x, and finally within the last namespace, row_three, we find the name we were looking for: book.

[ 35 ]

A Gentle Introduction to Python

Chapter 1

Walking through a namespace will be clearer when we'll be dealing with real code examples. For now, just keep in mind that namespaces are places where names are associated with objects. There is another concept, which is closely related to that of a namespace, which I'd like to briefly talk about: the scope.

Scopes According to Python's documentation: " A scope is a textual region of a Python program, where a namespace is directly accessible." Directly accessible means that when you're looking for an unqualified reference to a name, Python tries to find it in the namespace. Scopes are determined statically, but actually, during runtime, they are used dynamically. This means that by inspecting the source code, you can tell what the scope of an object is, but this doesn't prevent the software from altering that during runtime. There are four different scopes that Python makes accessible (not necessarily all of them are present at the same time, of course): The local scope, which is the innermost one and contains the local names. The enclosing scope, that is, the scope of any enclosing function. It contains non-local names and also non-global names. The global scope contains the global names. The built-in scope contains the built-in names. Python comes with a set of functions that you can use in an off-the-shelf fashion, such as print, all, abs, and so on. They live in the built-in scope. The rule is the following: when we refer to a name, Python starts looking for it in the current namespace. If the name is not found, Python continues the search to the enclosing scope and this continues until the built-in scope is searched. If a name hasn't been found after searching the built-in scope, then Python raises a NameError exception, which basically means that the name hasn't been defined (you saw this in the preceding example).

[ 36 ]

A Gentle Introduction to Python

Chapter 1

The order in which the namespaces are scanned when looking for a name is therefore: local, enclosing, global, built-in (LEGB). This is all very theoretical, so let's see an example. In order to show you local and enclosing namespaces, I will have to define a few functions. Don't worry if you are not familiar with their syntax for the moment. We'll study functions in Chapter 4, Functions, the Building Blocks of Code. Just remember that in the following code, when you see def, it means I'm defining a function: # scopes1.py # Local versus Global # we define a function, called local def local(): m = 7 print(m) m = 5 print(m) # we call, or `execute` the function local local()

In the preceding example, we define the same name m, both in the global scope and in the local one (the one defined by the local function). When we execute this program with the following command (have you activated your virtualenv?): $ python scopes1.py

We see two numbers printed on the console: 5 and 7. What happens is that the Python interpreter parses the file, top to bottom. First, it finds a couple of comment lines, which are skipped, then it parses the definition of the function local. When called, this function does two things: it sets up a name to an object representing number 7 and prints it. The Python interpreter keeps going and it finds another name binding. This time the binding happens in the global scope and the value is 5. The next line is a call to the print function, which is executed (and so we get the first value printed on the console: 5). After this, there is a call to the function local. At this point, Python executes the function, so at this time, the binding m = 7 happens and it's printed.

[ 37 ]

A Gentle Introduction to Python

Chapter 1

One very important thing to notice is that the part of the code that belongs to the definition of the local function is indented by four spaces on the right. Python, in fact, defines scopes by indenting the code. You walk into a scope by indenting, and walk out of it by unindenting. Some coders use two spaces, others three, but the suggested number of spaces to use is four. It's a good measure to maximize readability. We'll talk more about all the conventions you should embrace when writing Python code later. What would happen if we removed that m = 7 line? Remember the LEGB rule. Python would start looking for m in the local scope (function local), and, not finding it, it would go to the next enclosing scope. The next one, in this case, is the global one because there is no enclosing function wrapped around local. Therefore, we would see two numbers 5 printed on the console. Let's actually see what the code would look like: # scopes2.py # Local versus Global def local(): # m doesn't belong to the scope defined by the local function # so Python will keep looking into the next enclosing scope. # m is finally found in the global scope print(m, 'printing from the local scope') m = 5 print(m, 'printing from the global scope') local()

Running scopes2.py will print this: $ python scopes2.py 5 printing from the global scope 5 printing from the local scope

As expected, Python prints m the first time, then when the function local is called, m isn't found in its scope, so Python looks for it following the LEGB chain until m is found in the global scope. Let's see an example with an extra layer, the enclosing scope: # scopes3.py # Local, Enclosing and Global def enclosing_func(): m = 13

[ 38 ]

A Gentle Introduction to Python

Chapter 1

def local(): # m doesn't belong to the scope defined by the local # function so Python will keep looking into the next # enclosing scope. This time m is found in the enclosing # scope print(m, 'printing from the local scope') # calling the function local local() m = 5 print(m, 'printing from the global scope') enclosing_func()

Running scopes3.py will print on the console: $ python scopes3.py (5, 'printing from the global scope') (13, 'printing from the local scope')

As you can see, the print instruction from the function local is referring to m as before. m is still not defined within the function itself, so Python starts walking scopes following the LEGB order. This time m is found in the enclosing scope. Don't worry if this is still not perfectly clear for now. It will come to you as we go through the examples in the book. The Classes section of the Python tutorial (https:/​/ docs.​python.​org/​3/​tutorial/​classes.​html) has an interesting paragraph about scopes and namespaces. Make sure you read it at some point if you want a deeper understanding of the subject. Before we finish off this chapter, I would like to talk a bit more about objects. After all, basically everything in Python is an object, so I think they deserve a bit more attention.

Objects and classes When I introduced objects previously in the A proper introduction section of the chapter, I said that we use them to represent real-life objects. For example, we sell goods of any kind on the web nowadays and we need to be able to handle, store, and represent them properly. But objects are actually so much more than that. Most of what you will ever do, in Python, has to do with manipulating objects.

[ 39 ]

A Gentle Introduction to Python

Chapter 1

So, without going into too much detail (we'll do that in Chapter 6, OOP, Decorators, and Iterators), I want to give you the in a nutshell kind of explanation about classes and objects. We've already seen that objects are Python's abstraction for data. In fact, everything in Python is an object, infact numbers, strings (data structures that hold text), containers, collections, even functions. You can think of them as if they were boxes with at least three features: an ID (unique), a type, and a value. But how do they come to life? How do we create them? How do we write our own custom objects? The answer lies in one simple word: classes. Objects are, in fact, instances of classes. The beauty of Python is that classes are objects themselves, but let's not go down this road. It leads to one of the most advanced concepts of this language: metaclasses. For now, the best way for you to get the difference between classes and objects is by means of an example. Say a friend tells you, I bought a new bike! You immediately understand what she's talking about. Have you seen the bike? No. Do you know what color it is? Nope. The brand? Nope. Do you know anything about it? Nope. But at the same time, you know everything you need in order to understand what your friend meant when she told you she bought a new bike. You know that a bike has two wheels attached to a frame, a saddle, pedals, handlebars, brakes, and so on. In other words, even if you haven't seen the bike itself, you know the concept of bike. An abstract set of features and characteristics that together form something called bike. In computer programming, that is called a class. It's that simple. Classes are used to create objects. In fact, objects are said to be instances of classes. In other words, we all know what a bike is; we know the class. But then I have my own bike, which is an instance of the bike class. And my bike is an object with its own characteristics and methods. You have your own bike. Same class, but different instance. Every bike ever created in the world is an instance of the bike class. Let's see an example. We will write a class that defines a bike and then we'll create two bikes, one red and one blue. I'll keep the code very simple, but don't fret if you don't understand everything about it; all you need to care about at this moment is to understand the difference between a class and an object (or instance of a class): # bike.py # let's define the class Bike class Bike: def __init__(self, colour, frame_material):

[ 40 ]

A Gentle Introduction to Python

Chapter 1

self.colour = colour self.frame_material = frame_material def brake(self): print("Braking!") # let's create a couple of instances red_bike = Bike('Red', 'Carbon fiber') blue_bike = Bike('Blue', 'Steel') # let's inspect the objects we have, instances of the Bike class. print(red_bike.colour) # prints: Red print(red_bike.frame_material) # prints: Carbon fiber print(blue_bike.colour) # prints: Blue print(blue_bike.frame_material) # prints: Steel # let's brake! red_bike.brake()

# prints: Braking!

I hope by now I don't need to tell you to run the file every time, right? The filename is indicated in the first line of the code block. Just run $ python filename, and you'll be fine. But remember to have your virtualenv activated! So many interesting things to notice here. First things first; the definition of a class happens with the class statement. Whatever code comes after the class statement, and is indented, is called the body of the class. In our case, the last line that belongs to the class definition is the print("Braking!") one. After having defined the class, we're ready to create instances. You can see that the class body hosts the definition of two methods. A method is basically (and simplistically) a function that belongs to a class. The first method, __init__, is an initializer. It uses some Python magic to set up the objects with the values we pass when we create it. Every method that has leading and trailing double underscores, in Python, is called a magic method. Magic methods are used by Python for a multitude of different purposes; hence it's never a good idea to name a custom method using two leading and trailing underscores. This naming convention is best left to Python. The other method we defined, brake, is just an example of an additional method that we could call if we wanted to brake the bike. It contains just a print statement, of course; it's an example.

[ 41 ]

A Gentle Introduction to Python

Chapter 1

We created two bikes then. One has red color and a carbon fiber frame, and the other one has blue color and a steel frame. We pass those values upon creation. After creation, we print out the color property and frame type of the red bike, and the frame type of the blue one just as an example. We also call the brake method of the red_bike. One last thing to notice. You remember I told you that the set of attributes of an object is considered to be a namespace? I hope it's clearer what I meant now. You see that by getting to the frame_type property through different namespaces (red_bike, blue_bike), we obtain different values. No overlapping, no confusion. The dot (.) operator is of course the means we use to walk into a namespace, in the case of objects as well.

Guidelines on how to write good code Writing good code is not as easy as it seems. As I already said before, good code exposes a long list of qualities that is quite hard to put together. Writing good code is, to some extent, an art. Regardless of where on the path you will be happy to settle, there is something that you can embrace which will make your code instantly better: PEP 8. According to Wikipedia: "Python's development is conducted largely through the Python Enhancement Proposal (PEP) process. The PEP process is the primary mechanism for proposing major new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python." PEP 8 is perhaps the most famous of all PEPs. It lays out a simple but effective set of guidelines to define Python aesthetics so that we write beautiful Python code. If you take one suggestion out of this chapter, please let it be this: use it. Embrace it. You will thank me later. Coding today is no longer a check-in/check-out business. Rather, it's more of a social effort. Several developers collaborate on a piece of code through tools such as Git and Mercurial, and the result is code that is fathered by many different hands. Git and Mercurial are probably the distributed revision control systems that are most used today. They are essential tools designed to help teams of developers collaborate on the same software.

[ 42 ]

A Gentle Introduction to Python

Chapter 1

These days, more than ever, we need to have a consistent way of writing code, so that readability is maximized. When all developers of a company abide by PEP 8, it's not uncommon for any of them landing on a piece of code to think they wrote it themselves. It actually happens to me all the time (I always forget the code I write). This has a tremendous advantage: when you read code that you could have written yourself, you read it easily. Without a convention, every coder would structure the code the way they like most, or simply the way they were taught or are used to, and this would mean having to interpret every line according to someone else's style. It would mean having to lose much more time just trying to understand it. Thanks to PEP 8, we can avoid this. I'm such a fan of it that I won't sign off a code review if the code doesn't respect it. So, please take the time to study it; it's very important. In the examples in this book, I will try to respect it as much as I can. Unfortunately, I don't have the luxury of 79 characters (which is the maximum line length suggested by PEP 8), and I will have to cut down on blank lines and other things, but I promise you I'll try to lay out my code so that it's as readable as possible.

The Python culture Python has been adopted widely in all coding industries. It's used by many different companies for many different purposes, and it's also used in education (it's an excellent language for that purpose, because of its many qualities and the fact that it's easy to learn). One of the reasons Python is so popular today is that the community around it is vast, vibrant, and full of brilliant people. Many events are organized all over the world, mostly either around Python or its main web framework, Django. Python is open, and very often so are the minds of those who embrace it. Check out the community page on the Python website for more information and get involved! There is another aspect to Python which revolves around the notion of being Pythonic. It has to do with the fact that Python allows you to use some idioms that aren't found elsewhere, at least not in the same form or as easy to use (I feel quite claustrophobic when I have to code in a language which is not Python now). Anyway, over the years, this concept of being Pythonic has emerged and, the way I understand it, is something along the lines of doing things the way they are supposed to be done in Python.

[ 43 ]

A Gentle Introduction to Python

Chapter 1

To help you understand a little bit more about Python's culture and about being Pythonic, I will show you the Zen of Python. A lovely Easter egg that is very popular. Open up a Python console and type import this. What follows is the result of this line: >>> import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those!

There are two levels of reading here. One is to consider it as a set of guidelines that have been put down in a fun way. The other one is to keep it in mind, and maybe read it once in a while, trying to understand how it refers to something deeper: some Python characteristics that you will have to understand deeply in order to write Python the way it's supposed to be written. Start with the fun level, and then dig deeper. Always dig deeper.

A note on IDEs Just a few words about IDEs. To follow the examples in this book, you don't need one; any text editor will do fine. If you want to have more advanced features, such as syntax coloring and auto completion, you will have to fetch yourself an IDE. You can find a comprehensive list of open source IDEs (just Google Python IDEs) on the Python website. I personally use Sublime Text editor. It's free to try out and it costs just a few dollars. I have tried many IDEs in my life, but this is the one that makes me most productive.

[ 44 ]

A Gentle Introduction to Python

Chapter 1

Two important pieces of advice: Whatever IDE you choose to use, try to learn it well so that you can exploit its strengths, but don't depend on it. Exercise yourself to work with VIM (or any other text editor) once in a while; learn to be able to do some work on any platform, with any set of tools. Whatever text editor/IDE you use, when it comes to writing Python, indentation is four spaces. Don't use tabs, don't mix them with spaces. Use four spaces, not two, not three, not five. Just use four. The whole world works like that, and you don't want to become an outcast because you were fond of the three-space layout.

Summary In this chapter, we started to explore the world of programming and that of Python. We've barely scratched the surface, just a little, touching concepts that will be discussed later on in the book in greater detail. We talked about Python's main features, who is using it and for what, and what are the different ways in which we can write a Python program. In the last part of the chapter, we flew over the fundamental notions of namespaces, scopes, classes, and objects. We also saw how Python code can be organized using modules and packages. On a practical level, we learned how to install Python on our system, how to make sure we have the tools we need, pip and virtualenv, and we also created and activated our first virtual environment. This will allow us to work in a self-contained environment without the risk of compromising the Python system installation. Now you're ready to start this journey with me. All you need is enthusiasm, an activated virtual environment, this book, your fingers, and some coffee. Try to follow the examples; I'll keep them simple and short. If you put them under your fingertips, you will retain them much better than if you just read them. In the next chapter, we will explore Python's rich set of built-in data types. There's much to cover and much to learn!

[ 45 ]

2 Built-in Data Types "Data! Data! Data!" he cried impatiently. "I can't make bricks without clay." – Sherlock Holmes – The Adventure of the Copper Beeches Everything you do with a computer is managing data. Data comes in many different shapes and flavors. It's the music you listen to, the movies you stream, the PDFs you open. Even the source of the chapter you're reading at this very moment is just a file, which is data. Data can be simple, an integer number to represent an age, or complex, like an order placed on a website. It can be about a single object or about a collection of them. Data can even be about data, that is, metadata. Data that describes the design of other data structures or data that describes application data or its context. In Python, objects are abstraction for data, and Python has an amazing variety of data structures that you can use to represent data, or combine them to create your own custom data. In this chapter, we are going to cover the following: Python objects' structures Mutability and immutability Built-in data types: numbers, strings, sequences, collections, and mapping types The collections module Enumerations

Built-in Data Types

Chapter 2

Everything is an object Before we delve into the specifics, I want you to be very clear about objects in Python, so let's talk a little bit more about them. As we already said, everything in Python is an object. But what really happens when you type an instruction like age = 42 in a Python module? If you go to http://pythontutor.com/, you can type that instruction into a text box and get its visual representation. Keep this website in mind; it's very useful to consolidate your understanding of what goes on behind the scenes. So, what happens is that an object is created. It gets an id, the type is set to int (integer number), and the value to 42. A name age is placed in the global namespace, pointing to that object. Therefore, whenever we are in the global namespace, after the execution of that line, we can retrieve that object by simply accessing it through its name: age. If you were to move house, you would put all the knives, forks, and spoons in a box and label it cutlery. Can you see it's exactly the same concept? Here's a screenshot of what it may look like (you may have to tweak the settings to get to the same view):

So, for the rest of this chapter, whenever you read something such as name = some_value, think of a name placed in the namespace that is tied to the scope in which the instruction was written, with a nice arrow pointing to an object that has an id, a type, and a value. There is a little bit more to say about this mechanism, but it's much easier to talk about it over an example, so we'll get back to this later.

[ 47 ]

Built-in Data Types

Chapter 2

Mutable or immutable? That is the question A first fundamental distinction that Python makes on data is about whether or not the value of an object changes. If the value can change, the object is called mutable, while if the value cannot change, the object is called immutable. It is very important that you understand the distinction between mutable and immutable because it affects the code you write, so here's a question: >>> >>> 42 >>> >>> 43

age = 42 age age = 43 age

#A

In the preceding code, on the line #A, have I changed the value of age? Well, no. But now it's 43 (I hear you say...). Yes, it's 43, but 42 was an integer number, of the type int, which is immutable. So, what happened is really that on the first line, age is a name that is set to point to an int object, whose value is 42. When we type age = 43, what happens is that another object is created, of the type int and value 43 (also, the id will be different), and the name age is set to point to it. So, we didn't change that 42 to 43. We actually just pointed age to a different location: the new int object whose value is 43. Let's see the same code also printing the IDs: >>> age = 42 >>> id(age) 4377553168 >>> age = 43 >>> id(age) 4377553200

Notice that we print the IDs by calling the built-in id function. As you can see, they are different, as expected. Bear in mind that age points to one object at a time: 42 first, then 43. Never together. Now, let's see the same example using a mutable object. For this example, let's just use a Person object, that has a property age (don't worry about the class declaration for now; it's there only for completeness): >>> class Person(): ... def __init__(self, age): ... self.age = age

[ 48 ]

Built-in Data Types

Chapter 2

... >>> fab = Person(age=42) >>> fab.age 42 >>> id(fab) 4380878496 >>> id(fab.age) 4377553168 >>> fab.age = 25 # I wish! >>> id(fab) # will be the same 4380878496 >>> id(fab.age) # will be different 4377552624

In this case, I set up an object fab whose type is Person (a custom class). On creation, the object is given the age of 42. I'm printing it, along with the object id, and the ID of age as well. Notice that, even after I change age to be 25, the ID of fab stays the same (while the ID of age has changed, of course). Custom objects in Python are mutable (unless you code them not to be). Keep this concept in mind; it's very important. I'll remind you about it throughout the rest of the chapter.

Numbers Let's start by exploring Python's built-in data types for numbers. Python was designed by a man with a master's degree in mathematics and computer science, so it's only logical that it has amazing support for numbers. Numbers are immutable objects.

Integers Python integers have an unlimited range, subject only to the available virtual memory. This means that it doesn't really matter how big a number you want to store is: as long as it can fit in your computer's memory, Python will take care of it. Integer numbers can be positive, negative, and 0 (zero). They support all the basic mathematical operations, as shown in the following example: >>> >>> >>> 17 >>>

a = 14 b = 3 a + b # addition a - b

# subtraction

[ 49 ]

Built-in Data Types

Chapter 2

11 >>> a * b # multiplication 42 >>> a / b # true division 4.666666666666667 >>> a // b # integer division 4 >>> a % b # modulo operation (reminder of division) 2 >>> a ** b # power operation 2744

The preceding code should be easy to understand. Just notice one important thing: Python has two division operators, one performs the so-called true division (/), which returns the quotient of the operands, and the other one, the so-called integer division (//), which returns the floored quotient of the operands. It might be worth noting that in Python 2 the division operator / behaves differently than in Python 3. See how that is different for positive and negative numbers: >>> 7 / 4 # true division 1.75 >>> 7 // 4 # integer division, truncation returns 1 1 >>> -7 / 4 # true division again, result is opposite of previous -1.75 >>> -7 // 4 # integer div., result not the opposite of previous -2

This is an interesting example. If you were expecting a -1 on the last line, don't feel bad, it's just the way Python works. The result of an integer division in Python is always rounded towards minus infinity. If, instead of flooring, you want to truncate a number to an integer, you can use the built-in int function, as shown in the following example: >>> int(1.75) 1 >>> int(-1.75) -1

Notice that the truncation is done toward 0.

[ 50 ]

Built-in Data Types

Chapter 2

There is also an operator to calculate the remainder of a division. It's called a modulo operator, and it's represented by a percentage (%): >>> 10 % 3 1 >>> 10 % 4 2

# remainder of the division 10 // 3 # remainder of the division 10 // 4

One nice feature introduced in Python 3.6 is the ability to add underscores within number literals (between digits or base specifiers, but not leading or trailing). The purpose is to help make some numbers more readable, like for example 1_000_000_000: >>> n = 1_024 >>> n 1024 >>> hex_n = 0x_4_0_0 >>> hex_n 1024

# 0x400 == 1024

Booleans Boolean algebra is that subset of algebra in which the values of the variables are the truth values: true and false. In Python, True and False are two keywords that are used to represent truth values. Booleans are a subclass of integers, and behave respectively like 1 and 0. The equivalent of the int class for Booleans is the bool class, which returns either True or False. Every built-in Python object has a value in the Boolean context, which means they basically evaluate to either True or False when fed to the bool function. We'll see all about this in Chapter 3, Iterating and Making Decisions. Boolean values can be combined in Boolean expressions using the logical operators and, or, and not. Again, we'll see them in full in the next chapter, so for now let's just see a simple example: >>> int(True) 1 >>> int(False) 0 >>> bool(1) # True >>> bool(-42) True >>> bool(0) #

# True behaves like 1 # False behaves like 0 1 evaluates to True in a boolean context # and so does every non-zero number 0 evaluates to False

[ 51 ]

Built-in Data Types

Chapter 2

False >>> # quick peak at the operators (and, or, not) >>> not True False >>> not False True >>> True and True True >>> False or True True

You can see that True and False are subclasses of integers when you try to add them. Python upcasts them to integers and performs the addition: >>> 1 + True 2 >>> False + 42 42 >>> 7 - True 6

Upcasting is a type conversion operation that goes from a subclass to its parent. In the example presented here, True and False, which belong to a class derived from the integer class, are converted back to integers when needed. This topic is about inheritance and will be explained in detail in Chapter 6, OOP, Decorators, and Iterators.

Real numbers Real numbers, or floating point numbers, are represented in Python according to the IEEE 754 double-precision binary floating-point format, which is stored in 64 bits of information divided into three sections: sign, exponent, and mantissa. Quench your thirst for knowledge about this format on Wikipedia: http:/​/​en.​wikipedia.​org/​wiki/​Double-​precision_​floatingpoint_​format.

[ 52 ]

Built-in Data Types

Chapter 2

Usually, programming languages give coders two different formats: single and double precision. The former takes up 32 bits of memory, and the latter 64. Python supports only the double format. Let's see a simple example: >>> pi = 3.1415926536 # how many digits of PI can you remember? >>> radius = 4.5 >>> area = pi * (radius ** 2) >>> area 63.617251235400005

In the calculation of the area, I wrapped the radius ** 2 within braces. Even though that wasn't necessary because the power operator has higher precedence than the multiplication one, I think the formula reads more easily like that. Moreover, should you get a slightly different result for the area, don't worry. It might depend on your OS, how Python was compiled, and so on. As long as the first few decimal digits are correct, you know it's a correct result. The sys.float_info struct sequence holds information about how floating point numbers will behave on your system. This is what I see on my box: >>> import sys >>> sys.float_info sys.float_info(max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1)

Let's make a few considerations here: we have 64 bits to represent float numbers. This means we can represent at most 2 ** 64 == 18,446,744,073,709,551,616 numbers with that amount of bits. Take a look at the max and epsilon values for the float numbers, and you'll realize it's impossible to represent them all. There is just not enough space, so they are approximated to the closest representable number. You probably think that only extremely big or extremely small numbers suffer from this issue. Well, think again and try the following in your console: >>> 0.3 - 0.1 * 3 # this should be 0!!! -5.551115123125783e-17

[ 53 ]

Built-in Data Types

Chapter 2

What does this tell you? It tells you that double precision numbers suffer from approximation issues even when it comes to simple numbers like 0.1 or 0.3. Why is this important? It can be a big problem if you're handling prices, or financial calculations, or any kind of data that needs not to be approximated. Don't worry, Python gives you the decimal type, which doesn't suffer from these issues; we'll see them in a moment.

Complex numbers Python gives you complex numbers support out of the box. If you don't know what complex numbers are, they are numbers that can be expressed in the form a + ib where a and b are real numbers, and i (or j if you're an engineer) is the imaginary unit, that is, the square root of -1. a and b are called, respectively, the real and imaginary part of the number. It's actually unlikely you'll be using them, unless you're coding something scientific. Let's see a small example: >>> c = 3.14 + 2.73j >>> c.real # real part 3.14 >>> c.imag # imaginary part 2.73 >>> c.conjugate() # conjugate of A + Bj is A - Bj (3.14-2.73j) >>> c * 2 # multiplication is allowed (6.28+5.46j) >>> c ** 2 # power operation as well (2.4067000000000007+17.1444j) >>> d = 1 + 1j # addition and subtraction as well >>> c - d (2.14+1.73j)

Fractions and decimals Let's finish the tour of the number department with a look at fractions and decimals. Fractions hold a rational numerator and denominator in their lowest forms. Let's see a quick example: >>> from fractions import Fraction >>> Fraction(10, 6) # mad hatter? Fraction(5, 3) # notice it's been simplified >>> Fraction(1, 3) + Fraction(2, 3) # 1/3 + 2/3 == 3/3 == 1/1

[ 54 ]

Built-in Data Types

Chapter 2

Fraction(1, 1) >>> f = Fraction(10, 6) >>> f.numerator 5 >>> f.denominator 3

Although they can be very useful at times, it's not that common to spot them in commercial software. Much easier instead, is to see decimal numbers being used in all those contexts where precision is everything; for example, in scientific and financial calculations. It's important to remember that arbitrary precision decimal numbers come at a price in performance, of course. The amount of data to be stored for each number is far greater than it is for fractions or floats as well as the way they are handled, which causes the Python interpreter much more work behind the scenes. Another interesting thing to note is that you can get and set the precision by accessing decimal.getcontext().prec. Let's see a quick example with decimal numbers: >>> from decimal import Decimal as D # rename for brevity >>> D(3.14) # pi, from float, so approximation issues Decimal('3.140000000000000124344978758017532527446746826171875') >>> D('3.14') # pi, from a string, so no approximation issues Decimal('3.14') >>> D(0.1) * D(3) - D(0.3) # from float, we still have the issue Decimal('2.775557561565156540423631668E-17') >>> D('0.1') * D(3) - D('0.3') # from string, all perfect Decimal('0.0') >>> D('1.4').as_integer_ratio() # 7/5 = 1.4 (isn't this cool?!) (7, 5)

Notice that when we construct a Decimal number from a float, it takes on all the approximation issues float may come from. On the other hand, when the Decimal has no approximation issues (for example, when we feed an int or a string representation to the constructor), then the calculation has no quirky behavior. When it comes to money, use decimals. This concludes our introduction to built-in numeric types. Let's now look at sequences.

[ 55 ]

Built-in Data Types

Chapter 2

Immutable sequences Let's start with immutable sequences: strings, tuples, and bytes.

Strings and bytes Textual data in Python is handled with str objects, more commonly known as strings. They are immutable sequences of Unicode code points. Unicode code points can represent a character, but can also have other meanings, such as formatting data, for example. Python, unlike other languages, doesn't have a char type, so a single character is rendered simply by a string of length 1. Unicode is an excellent way to handle data, and should be used for the internals of any application. When it comes to storing textual data though, or sending it on the network, you may want to encode it, using an appropriate encoding for the medium you're using. The result of an encoding produces a bytes object, whose syntax and behavior is similar to that of strings. String literals are written in Python using single, double, or triple quotes (both single or double). If built with triple quotes, a string can span on multiple lines. An example will clarify this: >>> # 4 ways to make a string >>> str1 = 'This is a string. We built it with single quotes.' >>> str2 = "This is also a string, but built with double quotes." >>> str3 = '''This is built using triple quotes, ... so it can span multiple lines.''' >>> str4 = """This too ... is a multiline one ... built with triple double-quotes.""" >>> str4 #A 'This too\nis a multiline one\nbuilt with triple double-quotes.' >>> print(str4) #B This too is a multiline one built with triple double-quotes.

In #A and #B, we print str4, first implicitly, and then explicitly, using the print function. A nice exercise would be to find out why they are different. Are you up to the challenge? (hint: look up the str function.)

[ 56 ]

Built-in Data Types

Chapter 2

Strings, like any sequence, have a length. You can get this by calling the len function: >>> len(str1) 49

Encoding and decoding strings Using the encode/decode methods, we can encode Unicode strings and decode bytes objects. UTF-8 is a variable length character encoding, capable of encoding all possible Unicode code points. It is the dominant encoding for the web. Notice also that by adding a literal b in front of a string declaration, we're creating a bytes object: >>> s = "This is üŋíc0de" # unicode string: code points >>> type(s)

>>> encoded_s = s.encode('utf-8') # utf-8 encoded version of s >>> encoded_s b'This is \xc3\xbc\xc5\x8b\xc3\xadc0de' # result: bytes object >>> type(encoded_s) # another way to verify it

>>> encoded_s.decode('utf-8') # let's revert to the original 'This is üŋíc0de' >>> bytes_obj = b"A bytes object" # a bytes object >>> type(bytes_obj)

Indexing and slicing strings When manipulating sequences, it's very common to have to access them at one precise position (indexing), or to get a subsequence out of them (slicing). When dealing with immutable sequences, both operations are read-only. While indexing comes in one form, a zero-based access to any position within the sequence, slicing comes in different forms. When you get a slice of a sequence, you can specify the start and stop positions, and the step. They are separated with a colon (:) like this: my_sequence[start:stop:step]. All the arguments are optional, start is inclusive, and stop is exclusive. It's much easier to show an example, rather than explain them further in words: >>> s = "The trouble is you think you have time." >>> s[0] # indexing at position 0, which is the first char 'T' >>> s[5] # indexing at position 5, which is the sixth char 'r'

[ 57 ]

Built-in Data Types

Chapter 2

>>> s[:4] # slicing, we specify only the stop position 'The ' >>> s[4:] # slicing, we specify only the start position 'trouble is you think you have time.' >>> s[2:14] # slicing, both start and stop positions 'e trouble is' >>> s[2:14:3] # slicing, start, stop and step (every 3 chars) 'erb ' >>> s[:] # quick way of making a copy 'The trouble is you think you have time.'

Of all the lines, the last one is probably the most interesting. If you don't specify a parameter, Python will fill in the default for you. In this case, start will be the start of the string, stop will be the end of the string, and step will be the default 1. This is an easy and quick way of obtaining a copy of the string s (same value, but different object). Can you find a way to get the reversed copy of a string using slicing (don't look it up; find it for yourself)?

String formatting One of the features strings have is the ability to be used as a template. There are several different ways of formatting a string, and for the full list of possibilities, I encourage you to look up the documentation. Here are some common examples: >>> greet_old = 'Hello %s!' >>> greet_old % 'Fabrizio' 'Hello Fabrizio!' >>> greet_positional = 'Hello {} {}!' >>> greet_positional.format('Fabrizio', 'Romano') 'Hello Fabrizio Romano!' >>> greet_positional_idx = 'This is {0}! {1} loves {0}!' >>> greet_positional_idx.format('Python', 'Fabrizio') 'This is Python! Fabrizio loves Python!' >>> greet_positional_idx.format('Coffee', 'Fab') 'This is Coffee! Fab loves Coffee!' >>> keyword = 'Hello, my name is {name} {last_name}' >>> keyword.format(name='Fabrizio', last_name='Romano') 'Hello, my name is Fabrizio Romano'

[ 58 ]

Built-in Data Types

Chapter 2

In the previous example, you can see four different ways of formatting stings. The first one, which relies on the % operator, is deprecated and shouldn't be used any more. The current, modern way to format a string is by using the format string method. You can see, from the different examples, that a pair of curly braces acts as a placeholder within the string. When we call format, we feed it data that replaces the placeholders. We can specify indexes (and much more) within the curly braces, and even names, which implies we'll have to call format using keyword arguments instead of positional ones. Notice how greet_positional_idx is rendered differently by feeding different data to the call to format. Apparently, I'm into Python and coffee... big surprise! One last feature I want to show you is a relatively new addition to Python (Version 3.6) and it's called formatted string literals. This feature is quite cool: strings are prefixed with f, and contain replacement fields surrounded by curly braces. Replacement fields are expressions evaluated at runtime, and then formatted using the format protocol: >>> name = 'Fab' >>> age = 42 >>> f"Hello! My name is {name} and I'm {age}" "Hello! My name is Fab and I'm 42" >>> from math import pi >>> f"No arguing with {pi}, it's irrational..." "No arguing with 3.141592653589793, it's irrational..."

Check out the official documentation to learn everything about string formatting and how powerful it can be.

Tuples The last immutable sequence type we're going to see is the tuple. A tuple is a sequence of arbitrary Python objects. In a tuple, items are separated by commas. They are used everywhere in Python, because they allow for patterns that are hard to reproduce in other languages. Sometimes tuples are used implicitly; for example, to set up multiple variables on one line, or to allow a function to return multiple different objects (usually a function returns one object only, in many other languages), and even in the Python console, you can use tuples implicitly to print multiple elements with one single instruction. We'll see examples for all these cases: >>> t = () # empty tuple >>> type(t)

[ 59 ]

Built-in Data Types

Chapter 2

>>> one_element_tuple = (42, ) # you need the comma! >>> three_elements_tuple = (1, 3, 5) # braces are optional here >>> a, b, c = 1, 2, 3 # tuple for multiple assignment >>> a, b, c # implicit tuple to print with one instruction (1, 2, 3) >>> 3 in three_elements_tuple # membership test True

Notice that the membership operator in can also be used with lists, strings, dictionaries, and, in general, with collection and sequence objects. Notice that to create a tuple with one item, we need to put that comma after the item. The reason is that without the comma that item is just itself wrapped in braces, kind of in a redundant mathematical expression. Notice also that on assignment, braces are optional so my_tuple = 1, 2, 3 is the same as my_tuple = (1, 2, 3). One thing that tuple assignment allows us to do, is one-line swaps, with no need for a third temporary variable. Let's see first a more traditional way of doing it: >>> >>> >>> >>> >>> (2,

a, b = 1, 2 c = a # we need three lines and a temporary var c a = b b = c a, b # a and b have been swapped 1)

And now let's see how we would do it in Python: >>> >>> >>> (1,

a, b = 0, 1 a, b = b, a a, b 0)

# this is the Pythonic way to do it

[ 60 ]

Built-in Data Types

Chapter 2

Take a look at the line that shows you the Pythonic way of swapping two values. Do you remember what I wrote in Chapter 1, A Gentle Introduction to Python? A Python program is typically one-fifth to one-third the size of equivalent Java or C++ code, and features like one-line swaps contribute to this. Python is elegant, where elegance in this context also means economy. Because they are immutable, tuples can be used as keys for dictionaries (we'll see this shortly). To me, tuples are Python's built-in data that most closely represent a mathematical vector. This doesn't mean that this was the reason for which they were created though. Tuples usually contain an heterogeneous sequence of elements, while on the other hand, lists are most of the times homogeneous. Moreover, tuples are normally accessed via unpacking or indexing, while lists are usually iterated over.

Mutable sequences Mutable sequences differ from their immutable sisters in that they can be changed after creation. There are two mutable sequence types in Python: lists and byte arrays. I said before that the dictionary is the king of data structures in Python. I guess this makes the list its rightful queen.

Lists Python lists are mutable sequences. They are very similar to tuples, but they don't have the restrictions of immutability. Lists are commonly used to storing collections of homogeneous objects, but there is nothing preventing you from store heterogeneous collections as well. Lists can be created in many different ways. Let's see an example: >>> [] # empty list [] >>> list() # same as [] [] >>> [1, 2, 3] # as with tuples, items are comma separated [1, 2, 3] >>> [x + 5 for x in [2, 3, 4]] # Python is magic [7, 8, 9] >>> list((1, 3, 5, 7, 9)) # list from a tuple [1, 3, 5, 7, 9] >>> list('hello') # list from a string ['h', 'e', 'l', 'l', 'o']

[ 61 ]

Built-in Data Types

Chapter 2

In the previous example, I showed you how to create a list using different techniques. I would like you to take a good look at the line that says Python is magic, which I am not expecting you to fully understand at this point (unless you cheated and you're not a novice!). That is called a list comprehension, a very powerful functional feature of Python, which we'll see in detail in Chapter 5, Saving Time and Memory. I just wanted to make your mouth water at this point. Creating lists is good, but the real fun comes when we use them, so let's see the main methods they gift us with: >>> a = [1, 2, 1, 3] >>> a.append(13) # we can append anything at the end >>> a [1, 2, 1, 3, 13] >>> a.count(1) # how many `1` are there in the list? 2 >>> a.extend([5, 7]) # extend the list by another (or sequence) >>> a [1, 2, 1, 3, 13, 5, 7] >>> a.index(13) # position of `13` in the list (0-based indexing) 4 >>> a.insert(0, 17) # insert `17` at position 0 >>> a [17, 1, 2, 1, 3, 13, 5, 7] >>> a.pop() # pop (remove and return) last element 7 >>> a.pop(3) # pop element at position 3 1 >>> a [17, 1, 2, 3, 13, 5] >>> a.remove(17) # remove `17` from the list >>> a [1, 2, 3, 13, 5] >>> a.reverse() # reverse the order of the elements in the list >>> a [5, 13, 3, 2, 1] >>> a.sort() # sort the list >>> a [1, 2, 3, 5, 13] >>> a.clear() # remove all elements from the list >>> a []

The preceding code gives you a roundup of a list's main methods. I want to show you how powerful they are, using extend as an example. You can extend lists using any sequence type:

[ 62 ]

Built-in Data Types

Chapter 2

>>> a = list('hello') # makes a list from a string >>> a ['h', 'e', 'l', 'l', 'o'] >>> a.append(100) # append 100, heterogeneous type >>> a ['h', 'e', 'l', 'l', 'o', 100] >>> a.extend((1, 2, 3)) # extend using tuple >>> a ['h', 'e', 'l', 'l', 'o', 100, 1, 2, 3] >>> a.extend('...') # extend using string >>> a ['h', 'e', 'l', 'l', 'o', 100, 1, 2, 3, '.', '.', '.']

Now, let's see what are the most common operations you can do with lists: >>> >>> 1 >>> 7 >>> 16 >>> 4 >>> >>> [1, >>> [1,

a = [1, 3, 5, 7] min(a) # minimum value in the list max(a)

# maximum value in the list

sum(a)

# sum of all values in the list

len(a)

# number of elements in the list

b = [6, 7, 8] a + b # `+` with list means concatenation 3, 5, 7, 6, 7, 8] a * 2 # `*` has also a special meaning 3, 5, 7, 1, 3, 5, 7]

The last two lines in the preceding code are quite interesting because they introduce us to a concept called operator overloading. In short, it means that operators such as +, -. *, %, and so on, may represent different operations according to the context they are used in. It doesn't make any sense to sum two lists, right? Therefore, the + sign is used to concatenate them. Hence, the * sign is used to concatenate the list to itself according to the right operand. Now, let's take a step further and see something a little more interesting. I want to show you how powerful the sorted method can be and how easy it is in Python to achieve results that require a great deal of effort in other languages: >>> from operator import itemgetter >>> a = [(5, 3), (1, 3), (1, 2), (2, -1), (4, 9)] >>> sorted(a) [(1, 2), (1, 3), (2, -1), (4, 9), (5, 3)] >>> sorted(a, key=itemgetter(0)) [(1, 3), (1, 2), (2, -1), (4, 9), (5, 3)]

[ 63 ]

Built-in Data Types

Chapter 2

>>> sorted(a, key=itemgetter(0, 1)) [(1, 2), (1, 3), (2, -1), (4, 9), (5, 3)] >>> sorted(a, key=itemgetter(1)) [(2, -1), (1, 2), (5, 3), (1, 3), (4, 9)] >>> sorted(a, key=itemgetter(1), reverse=True) [(4, 9), (5, 3), (1, 3), (1, 2), (2, -1)]

The preceding code deserves a little explanation. First of all, a is a list of tuples. This means each element in a is a tuple (a 2-tuple, to be precise). When we call sorted(some_list), we get a sorted version of some_list. In this case, the sorting on a 2-tuple works by sorting them on the first item in the tuple, and on the second when the first one is the same. You can see this behavior in the result of sorted(a), which yields [(1, 2), (1, 3), ...]. Python also gives us the ability to control which element(s) of the tuple the sorting must be run against. Notice that when we instruct the sorted function to work on the first element of each tuple (by key=itemgetter(0)), the result is different: [(1, 3), (1, 2), ...]. The sorting is done only on the first element of each tuple (which is the one at position 0). If we want to replicate the default behavior of a simple sorted(a) call, we need to use key=itemgetter(0, 1), which tells Python to sort first on the elements at position 0 within the tuples, and then on those at position 1. Compare the results and you'll see they match. For completeness, I included an example of sorting only on the elements at position 1, and the same but in reverse order. If you have ever seen sorting in Java, I expect you to be quite impressed at this moment. The Python sorting algorithm is very powerful, and it was written by Tim Peters (we've already seen this name, can you recall when?). It is aptly named Timsort, and it is a blend between merge and insertion sort and has better time performances than most other algorithms used for mainstream programming languages. Timsort is a stable sorting algorithm, which means that when multiple records have the same key, their original order is preserved. We've seen this in the result of sorted(a, key=itemgetter(0)), which has yielded [(1, 3), (1, 2), ...], in which the order of those two tuples has been preserved because they have the same value at position 0.

[ 64 ]

Built-in Data Types

Chapter 2

Byte arrays To conclude our overview of mutable sequence types, let's spend a couple of minutes on the bytearray type. Basically, they represent the mutable version of bytes objects. They expose most of the usual methods of mutable sequences as well as most of the methods of the bytes type. Items are integers in the range [0, 256). When it comes to intervals, I'm going to use the standard notation for open/closed ranges. A square bracket on one end means that the value is included, while a round brace means it's excluded. The granularity is usually inferred by the type of the edge elements so, for example, the interval [3, 7] means all integers between 3 and 7, inclusive. On the other hand, (3, 7) means all integers between 3 and 7 exclusive (hence 4, 5, and 6). Items in a bytearray type are integers between 0 and 256; 0 is included, 256 is not. One reason intervals are often expressed like this is to ease coding. If we break a range [a, b) into N consecutive ranges, we can easily represent the original one as a concatenation like this: [a,k1)+[k1,k2)+[k2,k3)+...+[kN-1,b) The middle points (ki) being excluded on one end, and included on the other end, allow for easy concatenation and splitting when intervals are handled in the code. Let's see a quick example with the bytearray type: >>> bytearray() # empty bytearray object bytearray(b'') >>> bytearray(10) # zero-filled instance with given length bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00') >>> bytearray(range(5)) # bytearray from iterable of integers bytearray(b'\x00\x01\x02\x03\x04') >>> name = bytearray(b'Lina') #A - bytearray from bytes >>> name.replace(b'L', b'l') bytearray(b'lina') >>> name.endswith(b'na') True >>> name.upper() bytearray(b'LINA') >>> name.count(b'L') 1

[ 65 ]

Built-in Data Types

Chapter 2

As you can see in the preceding code, there are a few ways to create a bytearray object. They can be useful in many situations; for example, when receiving data through a socket, they eliminate the need to concatenate data while polling, hence they can prove to be very handy. On the line #A, I created a bytearray named as name from the bytes literal b'Lina' to show you how the bytearray object exposes methods from both sequences and strings, which is extremely handy. If you think about it, they can be considered as mutable strings.

Set types Python also provides two set types, set and frozenset. The set type is mutable, while frozenset is immutable. They are unordered collections of immutable objects. Hashability is a characteristic that allows an object to be used as a set member as well as a key for a dictionary, as we'll see very soon. From the official documentation: An object is hashable if it has a hash value which never changes during its lifetime, and can be compared to other objects. Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally. All of Python’s immutable built-in objects are hashable while mutable containers are not. Objects that compare equally must have the same hash value. Sets are very commonly used to test for membership, so let's introduce the in operator in the following example: >>> small_primes = set() # empty set >>> small_primes.add(2) # adding one element at a time >>> small_primes.add(3) >>> small_primes.add(5) >>> small_primes {2, 3, 5} >>> small_primes.add(1) # Look what I've done, 1 is not a prime! >>> small_primes {1, 2, 3, 5} >>> small_primes.remove(1) # so let's remove it >>> 3 in small_primes # membership test True >>> 4 in small_primes False >>> 4 not in small_primes # negated membership test True >>> small_primes.add(3) # trying to add 3 again

[ 66 ]

Built-in Data Types >>> {2, >>> >>> {2, >>> {5} >>> {2,

Chapter 2

small_primes 3, 5} # no change, duplication is not allowed bigger_primes = set([5, 7, 11, 13]) # faster creation small_primes | bigger_primes # union operator `|` 3, 5, 7, 11, 13} small_primes & bigger_primes # intersection operator `&` small_primes - bigger_primes 3}

# difference operator `-`

In the preceding code, you can see two different ways to create a set. One creates an empty set and then adds elements one at a time. The other creates the set using a list of numbers as an argument to the constructor, which does all the work for us. Of course, you can create a set from a list or tuple (or any iterable) and then you can add and remove members from the set as you please. We'll look at iterable objects and iteration in the next chapter. For now, just know that iterable objects are objects you can iterate on in a direction. Another way of creating a set is by simply using the curly braces notation, like this: >>> small_primes = {2, 3, 5, 5, 3} >>> small_primes {2, 3, 5}

Notice I added some duplication to emphasize that the resulting set won't have any. Let's see an example about the immutable counterpart of the set type, frozenset: >>> small_primes = frozenset([2, 3, 5, 7]) >>> bigger_primes = frozenset([5, 7, 11]) >>> small_primes.add(11) # we cannot add to a frozenset Traceback (most recent call last): File "", line 1, in AttributeError: 'frozenset' object has no attribute 'add' >>> small_primes.remove(2) # neither we can remove Traceback (most recent call last): File "", line 1, in AttributeError: 'frozenset' object has no attribute 'remove' >>> small_primes & bigger_primes # intersect, union, etc. allowed frozenset({5, 7})

As you can see, frozenset objects are quite limited in respect of their mutable counterpart. They still prove very effective for membership test, union, intersection, and difference operations, and for performance reasons.

[ 67 ]

Built-in Data Types

Chapter 2

Mapping types – dictionaries Of all the built-in Python data types, the dictionary is easily the most interesting one. It's the only standard mapping type, and it is the backbone of every Python object. A dictionary maps keys to values. Keys need to be hashable objects, while values can be of any arbitrary type. Dictionaries are mutable objects. There are quite a few different ways to create a dictionary, so let me give you a simple example of how to create a dictionary equal to {'A': 1, 'Z': -1} in five different ways: >>> a >>> b >>> c >>> d >>> e >>> a True

= dict(A=1, Z=-1) = {'A': 1, 'Z': -1} = dict(zip(['A', 'Z'], [1, -1])) = dict([('A', 1), ('Z', -1)]) = dict({'Z': -1, 'A': 1}) == b == c == d == e # are they all the same? # They are indeed

Have you noticed those double equals? Assignment is done with one equal, while to check whether an object is the same as another one (or five in one go, in this case), we use double equals. There is also another way to compare objects, which involves the is operator, and checks whether the two objects are the same (if they have the same ID, not just the value), but unless you have a good reason to use it, you should use the double equals instead. In the preceding code, I also used one nice function: zip. It is named after the real-life zip, which glues together two things taking one element from each at a time. Let me show you an example: >>> list(zip(['h', 'e', 'l', 'l', 'o'], [1, 2, 3, 4, 5])) [('h', 1), ('e', 2), ('l', 3), ('l', 4), ('o', 5)] >>> list(zip('hello', range(1, 6))) # equivalent, more Pythonic [('h', 1), ('e', 2), ('l', 3), ('l', 4), ('o', 5)]

In the preceding example, I have created the same list in two different ways, one more explicit, and the other a little bit more Pythonic. Forget for a moment that I had to wrap the list constructor around the zip call (the reason is because zip returns an iterator, not a list, so if I want to see the result I need to exhaust that iterator into something—a list in this case), and concentrate on the result. See how zip has coupled the first elements of its two arguments together, then the second ones, then the third ones, and so on and so forth? Take a look at your pants (or at your purse, if you're a lady) and you'll see the same behavior in your actual zip. But let's go back to dictionaries and see how many wonderful methods they expose for allowing us to manipulate them as we want.

[ 68 ]

Built-in Data Types

Chapter 2

Let's start with the basic operations: >>> d = {} >>> d['a'] = 1 # let's set a couple of (key, value) pairs >>> d['b'] = 2 >>> len(d) # how many pairs? 2 >>> d['a'] # what is the value of 'a'? 1 >>> d # how does `d` look now? {'a': 1, 'b': 2} >>> del d['a'] # let's remove `a` >>> d {'b': 2} >>> d['c'] = 3 # let's add 'c': 3 >>> 'c' in d # membership is checked against the keys True >>> 3 in d # not the values False >>> 'e' in d False >>> d.clear() # let's clean everything from this dictionary >>> d {}

Notice how accessing keys of a dictionary, regardless of the type of operation we're performing, is done through square brackets. Do you remember strings, lists, and tuples? We were accessing elements at some position through square brackets as well, which is yet another example of Python's consistency. Let's see now three special objects called dictionary views: keys, values, and items. These objects provide a dynamic view of the dictionary entries and they change when the dictionary changes. keys() returns all the keys in the dictionary, values() returns all the values in the dictionary, and items() returns all the (key, value) pairs in the dictionary. According to the Python documentation: "Keys and values are iterated over in an arbitrary order which is non-random, varies across Python implementations, and depends on the dictionary’s history of insertions and deletions. If keys, values and items views are iterated over with no intervening modifications to the dictionary, the order of items will directly correspond."

[ 69 ]

Built-in Data Types

Chapter 2

Enough with this chatter; let's put all this down into code: >>> d = dict(zip('hello', range(5))) >>> d {'h': 0, 'e': 1, 'l': 3, 'o': 4} >>> d.keys() dict_keys(['h', 'e', 'l', 'o']) >>> d.values() dict_values([0, 1, 3, 4]) >>> d.items() dict_items([('h', 0), ('e', 1), ('l', 3), ('o', 4)]) >>> 3 in d.values() True >>> ('o', 4) in d.items() True

There are a few things to notice in the preceding code. First, notice how we're creating a dictionary by iterating over the zipped version of the string 'hello' and the list [0, 1, 2, 3, 4]. The string 'hello' has two 'l' characters inside, and they are paired up with the values 2 and 3 by the zip function. Notice how in the dictionary, the second occurrence of the 'l' key (the one with value 3), overwrites the first one (the one with value 2). Another thing to notice is that when asking for any view, the original order is now preserved, while before Version 3.6 there was no guarantee of that. As of Python 3.6, the dict type has been reimplemented to use a more compact representation. This resulted in dictionaries using 20% to 25% less memory when compared to Python 3.5. Moreover, in Python 3.6, as a side effect, dictionaries are natively ordered. This feature has received such a welcome from the community that in 3.7 it has become a legit feature of the language rather than an implementation side effect. A dict is ordered if it remembers the order in which keys were first inserted. We'll see how these views are fundamental tools when we talk about iterating over collections. Let's take a look now at some other methods exposed by Python's dictionaries; there's plenty of them and they are very useful: >>> d {'e': 1, 'h': 0, 'o': 4, 'l': 3} >>> d.popitem() # removes a random item (useful in algorithms) ('o', 4) >>> d {'h': 0, 'e': 1, 'l': 3} >>> d.pop('l') # remove item with key `l`

[ 70 ]

Built-in Data Types

Chapter 2

3 >>> d.pop('not-a-key') # remove a key not in dictionary: KeyError Traceback (most recent call last): File "", line 1, in KeyError: 'not-a-key' >>> d.pop('not-a-key', 'default-value') # with a default value? 'default-value' # we get the default value >>> d.update({'another': 'value'}) # we can update dict this way >>> d.update(a=13) # or this way (like a function call) >>> d {'h': 0, 'e': 1, 'another': 'value', 'a': 13} >>> d.get('a') # same as d['a'] but if key is missing no KeyError 13 >>> d.get('a', 177) # default value used if key is missing 13 >>> d.get('b', 177) # like in this case 177 >>> d.get('b') # key is not there, so None is returned

All these methods are quite simple to understand, but it's worth talking about that None, for a moment. Every function in Python returns None, unless the return statement is explicitly used to return something else, but we'll see this when we explore functions. None is frequently used to represent the absence of a value, and it is quite commonly used as a default value for arguments in function declaration. Some inexperienced coders sometimes write code that returns either False or None. Both False and None evaluate to False in a Boolean context so it may seem there is not much difference between them. But actually, I would argue there is quite an important difference: False means that we have information, and the information we have is False. None means no information. And no information is very different from information that is False. In layman's terms, if you ask your mechanic, Is my car ready?, there is a big difference between the answer, No, it's not (False) and, I have no idea (None). One last method I really like about dictionaries is setdefault. It behaves like get, but also sets the key with the given value if it is not there. Let's see an example: >>> d = {} >>> d.setdefault('a', 1) # 'a' is missing, we get default value 1 >>> d {'a': 1} # also, the key/value pair ('a', 1) has now been added >>> d.setdefault('a', 5) # let's try to override the value 1 >>> d {'a': 1} # no override, as expected

[ 71 ]

Built-in Data Types

Chapter 2

So, we're now at the end of this tour. Test your knowledge about dictionaries by trying to foresee what d looks like after this line: >>> d = {} >>> d.setdefault('a', {}).setdefault('b', []).append(1)

Don't worry if you don't get it immediately. I just wanted to encourage you to experiment with dictionaries. This concludes our tour of built-in data types. Before I discuss some considerations about what we've seen in this chapter, I want to take a peek briefly at the collections module.

The collections module When Python general purpose built-in containers (tuple, list, set, and dict) aren't enough, we can find specialized container datatypes in the collections module. They are: Data type

Description

namedtuple() Factory function for creating tuple subclasses with named fields

List-like container with fast appends and pops on either end ChainMap Dictionary-like class for creating a single view of multiple mappings Counter Dictionary subclass for counting hashable objects OrderedDict Dictionary subclass that remembers the order entries were added Dictionary subclass that calls a factory function to supply missing defaultdict values UserDict Wrapper around dictionary objects for easier dictionary subclassing UserList Wrapper around list objects for easier list subclassing UserString Wrapper around string objects for easier string subclassing deque

We don't have the room to cover all of them, but you can find plenty of examples in the official documentation, so here I'll just give a small example to show you namedtuple, defaultdict, and ChainMap.

[ 72 ]

Built-in Data Types

Chapter 2

namedtuple A namedtuple is a tuple-like object that has fields accessible by attribute lookup as well as being indexable and iterable (it's actually a subclass of tuple). This is sort of a compromise between a full-fledged object and a tuple, and it can be useful in those cases where you don't need the full power of a custom object, but you want your code to be more readable by avoiding weird indexing. Another use case is when there is a chance that items in the tuple need to change their position after refactoring, forcing the coder to refactor also all the logic involved, which can be very tricky. As usual, an example is better than a thousand words (or was it a picture?). Say we are handling data about the left and right eyes of a patient. We save one value for the left eye (position 0) and one for the right eye (position 1) in a regular tuple. Here's how that might be: >>> vision = (9.5, 8.8) >>> vision (9.5, 8.8) >>> vision[0] # left eye (implicit positional reference) 9.5 >>> vision[1] # right eye (implicit positional reference) 8.8

Now let's pretend we handle vision objects all the time, and at some point the designer decides to enhance them by adding information for the combined vision, so that a vision object stores data in this format: (left eye, combined, right eye). Do you see the trouble we're in now? We may have a lot of code that depends on vision[0] being the left eye information (which it still is) and vision[1] being the right eye information (which is no longer the case). We have to refactor our code wherever we handle these objects, changing vision[1] to vision[2], and it can be painful. We could have probably approached this a bit better from the beginning, by using a namedtuple. Let me show you what I mean: >>> >>> >>> >>> 9.5 >>> 9.5 >>> 8.8

from collections import namedtuple Vision = namedtuple('Vision', ['left', 'right']) vision = Vision(9.5, 8.8) vision[0] vision.left vision.right

# same as vision[0], but explicit # same as vision[1], but explicit

[ 73 ]

Built-in Data Types

Chapter 2

If within our code, we refer to the left and right eyes using vision.left and vision.right, all we need to do to fix the new design issue is to change our factory and the way we create instances. The rest of the code won't need to change: >>> >>> >>> 9.5 >>> 8.8 >>> 9.2

Vision = namedtuple('Vision', ['left', 'combined', 'right']) vision = Vision(9.5, 9.2, 8.8) vision.left # still correct vision.right

# still correct (though now is vision[2])

vision.combined

# the new vision[1]

You can see how convenient it is to refer to those values by name rather than by position. After all, a wise man once wrote, Explicit is better than implicit (can you recall where? Think Zen if you can't...). This example may be a little extreme; of course, it's not likely that our code designer will go for a change like this, but you'd be amazed to see how frequently issues similar to this one happen in a professional environment, and how painful it is to refactor them.

defaultdict The defaultdict data type is one of my favorites. It allows you to avoid checking if a key is in a dictionary by simply inserting it for you on your first access attempt, with a default value whose type you pass on creation. In some cases, this tool can be very handy and shorten your code a little. Let's see a quick example. Say we are updating the value of age, by adding one year. If age is not there, we assume it was 0 and we update it to 1: >>> d = {} >>> d['age'] = d.get('age', 0) + 1 >>> d {'age': 1} >>> d = {'age': 39} >>> d['age'] = d.get('age', 0) + 1 >>> d {'age': 40}

# age not there, we get 0 + 1

# age is there, we get 40

[ 74 ]

Built-in Data Types

Chapter 2

Now let's see how it would work with a defaultdict data type. The second line is actually the short version of a four-lines-long if clause that we would have to write if dictionaries didn't have the get method (we'll see all about if clauses in Chapter 3, Iterating and Making Decisions): >>> from collections import defaultdict >>> dd = defaultdict(int) # int is the default type (0 the value) >>> dd['age'] += 1 # short for dd['age'] = dd['age'] + 1 >>> dd defaultdict(, {'age': 1}) # 1, as expected

Notice how we just need to instruct the defaultdict factory that we want an int number to be used in case the key is missing (we'll get 0, which is the default for the int type). Also, notice that even though in this example there is no gain on the number of lines, there is definitely a gain in readability, which is very important. You can also use a different technique to instantiate a defaultdict data type, which involves creating a factory object. To dig deeper, please refer to the official documentation.

ChainMap ChainMap is an extremely nice data type which was introduced in Python 3.3. It

behaves like a normal dictionary but according to the Python documentation: "is provided for quickly linking a number of mappings so they can be treated as a single unit". This is usually much faster than creating one dictionary and running multiple update calls on it. ChainMap can be used to simulate nested scopes and is useful in templating. The underlying mappings are stored in a list. That list is public and can be accessed or updated using the maps attribute. Lookups search the underlying mappings successively until a key is found. By contrast, writes, updates, and deletions only operate on the first mapping. A very common use case is providing defaults, so let's see an example: >>> from collections import ChainMap >>> default_connection = {'host': 'localhost', 'port': 4567} >>> connection = {'port': 5678} >>> conn = ChainMap(connection, default_connection) # map creation >>> conn['port'] # port is found in the first dictionary 5678 >>> conn['host'] # host is fetched from the second dictionary 'localhost' >>> conn.maps # we can see the mapping objects [{'port': 5678}, {'host': 'localhost', 'port': 4567}]

[ 75 ]

Built-in Data Types

Chapter 2

>>> conn['host'] = 'packtpub.com' # let's add host >>> conn.maps [{'port': 5678, 'host': 'packtpub.com'}, {'host': 'localhost', 'port': 4567}] >>> del conn['port'] # let's remove the port information >>> conn.maps [{'host': 'packtpub.com'}, {'host': 'localhost', 'port': 4567}] >>> conn['port'] # now port is fetched from the second dictionary 4567 >>> dict(conn) # easy to merge and convert to regular dictionary {'host': 'packtpub.com', 'port': 4567}

I just love how Python makes your life easy. You work on a ChainMap object, configure the first mapping as you want, and when you need a complete dictionary with all the defaults as well as the customized items, you just feed the ChainMap object to a dict constructor. If you have never coded in other languages, such as Java or C++, you probably won't be able to appreciate fully how precious this is, and how Python makes your life so much easier. I do, I feel claustrophobic every time I have to code in some other language.

Enums Technically not a built-in data type, as you have to import them from the enum module, but definitely worth mentioning, are enumerations. They were introduced in Python 3.4, and though it is not that common to see them in professional code (yet), I thought I'd give you an example anyway. The official definition goes like this: "An enumeration is a set of symbolic names (members) bound to unique, constant values. Within an enumeration, the members can be compared by identity, and the enumeration itself can be iterated over." Say you need to represent traffic lights. In your code, you might resort to doing this: >>> >>> >>> >>> >>> >>>

GREEN = 1 YELLOW = 2 RED = 4 TRAFFIC_LIGHTS = (GREEN, YELLOW, RED) # or with a dict traffic_lights = {'GREEN': 1, 'YELLOW': 2, 'RED': 4}

[ 76 ]

Built-in Data Types

Chapter 2

There's nothing special about the preceding code. It's something, in fact, that is very common to find. But, consider doing this instead: >>> from enum import Enum >>> class TrafficLight(Enum): ... GREEN = 1 ... YELLOW = 2 ... RED = 4 ... >>> TrafficLight.GREEN

>>> TrafficLight.GREEN.name 'GREEN' >>> TrafficLight.GREEN.value 1 >>> TrafficLight(1)

>>> TrafficLight(4)

Ignoring for a moment the (relative) complexity of a class definition, you can appreciate how this might be more advantageous. The data structure is much cleaner, and the API it provides is much more powerful. I encourage you to check out the official documentation to explore all the great features you can find in the enum module. I think it's worth exploring, at least once.

Final considerations That's it. Now you have seen a very good proportion of the data structures that you will use in Python. I encourage you to take a dive into the Python documentation and experiment further with each and every data type we've seen in this chapter. It's worth it, believe me. Everything you'll write will be about handling data, so make sure your knowledge about it is rock solid. Before we leap into Chapter 3, Iterating and Making Decisions, I'd like to share some final considerations about different aspects that to my mind are important and not to be neglected.

[ 77 ]

Built-in Data Types

Chapter 2

Small values caching When we discussed objects at the beginning of this chapter, we saw that when we assigned a name to an object, Python creates the object, sets its value, and then points the name to it. We can assign different names to the same value and we expect different objects to be created, like this: >>> a = 1000000 >>> b = 1000000 >>> id(a) == id(b) False

In the preceding example, a and b are assigned to two int objects, which have the same value but they are not the same object, as you can see, their id is not the same. So let's do it again: >>> a = 5 >>> b = 5 >>> id(a) == id(b) True

Oh, oh! Is Python broken? Why are the two objects the same now? We didn't do a = b = 5, we set them up separately. Well, the answer is performances. Python caches short strings and small numbers, to avoid having many copies of them clogging up the system memory. Everything is handled properly under the hood so you don't need to worry a bit, but make sure that you remember this behavior should your code ever need to fiddle with IDs.

How to choose data structures As we've seen, Python provides you with several built-in data types and sometimes, if you're not that experienced, choosing the one that serves you best can be tricky, especially when it comes to collections. For example, say you have many dictionaries to store, each of which represents a customer. Within each customer dictionary, there's an 'id': 'code' unique identification code. In what kind of collection would you place them? Well, unless I know more about these customers, it's very hard to answer. What kind of access will I need? What sort of operations will I have to perform on each of them, and how many times? Will the collection change over time? Will I need to modify the customer dictionaries in any way? What is going to be the most frequent operation I will have to perform on the collection?

[ 78 ]

Built-in Data Types

Chapter 2

If you can answer the preceding questions, then you will know what to choose. If the collection never shrinks or grows (in other words, it won't need to add/delete any customer object after creation) or shuffles, then tuples are a possible choice. Otherwise, lists are a good candidate. Every customer dictionary has a unique identifier though, so even a dictionary could work. Let me draft these options for you: # example customer objects customer1 = {'id': 'abc123', 'full_name': 'Master Yoda'} customer2 = {'id': 'def456', 'full_name': 'Obi-Wan Kenobi'} customer3 = {'id': 'ghi789', 'full_name': 'Anakin Skywalker'} # collect them in a tuple customers = (customer1, customer2, customer3) # or collect them in a list customers = [customer1, customer2, customer3] # or maybe within a dictionary, they have a unique id after all customers = { 'abc123': customer1, 'def456': customer2, 'ghi789': customer3, }

Some customers we have there, right? I probably wouldn't go with the tuple option, unless I wanted to highlight that the collection is not going to change. I'd say usually a list is better, as it allows for more flexibility. Another factor to keep in mind is that tuples and lists are ordered collections. If you use a dictionary (prior to Python 3.6) or a set, you lose the ordering, so you need to know if ordering is important in your application. What about performances? For example, in a list, operations such as insertion and membership can take O(n), while they are O(1) for a dictionary. It's not always possible to use dictionaries though, if we don't have the guarantee that we can uniquely identify each item of the collection by means of one of its properties, and that the property in question is hashable (so it can be a key in dict).

[ 79 ]

Built-in Data Types

Chapter 2

If you're wondering what O(n) and O(1) mean, please Google big O notation. In this context, let's just say that if performing an operation Op on a data structure takes O(f(n)), it would mean that Op takes at most a time t ≤ c * f(n) to complete, where c is some positive constant, n is the size of the input, and f is some function. So, think of O(...) as an upper bound for the running time of an operation (it can be used also to size other measurable quantities, of course). Another way of understanding if you have chosen the right data structure is by looking at the code you have to write in order to manipulate it. If everything comes easily and flows naturally, then you probably have chosen correctly, but if you find yourself thinking your code is getting unnecessarily complicated, then you probably should try and decide whether you need to reconsider your choices. It's quite hard to give advice without a practical case though, so when you choose a data structure for your data, try to keep ease of use and performance in mind and give precedence to what matters most in the context you are in.

About indexing and slicing At the beginning of this chapter, we saw slicing applied on strings. Slicing, in general, applies to a sequence: tuples, lists, strings, and so on. With lists, slicing can also be used for assignment. I've almost never seen this used in professional code, but still, you know you can. Could you slice dictionaries or sets? I hear you scream, Of course not!. Excellent; I see we're on the same page here, so let's talk about indexing. There is one characteristic about Python indexing I haven't mentioned before. I'll show you by way of an example. How do you address the last element of a collection? Let's see: >>> >>> [0, >>> 10 >>> 9 >>> 9 >>>

a = list(range(10)) # `a` has 10 elements. Last one is 9. a 1, 2, 3, 4, 5, 6, 7, 8, 9] len(a) # its length is 10 elements a[len(a) - 1]

# position of last one is len(a) - 1

a[-1]

# but we don't need len(a)! Python rocks!

a[-2]

# equivalent to len(a) - 2

[ 80 ]

Built-in Data Types 8 >>> a[-3] 7

Chapter 2

# equivalent to len(a) - 3

If the list a has 10 elements, because of the 0-index positioning system of Python, the first one is at position 0 and the last one is at position 9. In the preceding example, the elements are conveniently placed in a position equal to their value: 0 is at position 0, 1 at position 1, and so on. So, in order to fetch the last element, we need to know the length of the whole list (or tuple, or string, and so on) and then subtract 1. Hence: len(a) - 1. This is so common an operation that Python provides you with a way to retrieve elements using negative indexing. This proves very useful when you do data manipulation. Here's a nice diagram about how indexing works on the string "HelloThere" (which is Obi-Wan Kenobi sarcastically greeting General Grievous):

Trying to address indexes greater than 9 or smaller than -10 will raise an IndexError, as expected.

About the names You may have noticed that, in order to keep the examples as short as possible, I have called many objects using simple letters, like a, b, c, d, and so on. This is perfectly OK when you debug on the console or when you show that a + b == 7, but it's bad practice when it comes to professional coding (or any type of coding, for that matter). I hope you will indulge me if I sometimes do it; the reason is to present the code in a more compact way.

[ 81 ]

Built-in Data Types

Chapter 2

In a real environment though, when you choose names for your data, you should choose them carefully and they should reflect what the data is about. So, if you have a collection of Customer objects, customers is a perfectly good name for it. Would customers_list, customers_tuple, or customers_collection work as well? Think about it for a second. Is it good to tie the name of the collection to the datatype? I don't think so, at least in most cases. So I'd say if you have an excellent reason to do so, go ahead; otherwise, don't. The reason is, once that customers_tuple starts being used in different places of your code, and you realize you actually want to use a list instead of a tuple, you're up for some fun refactoring (also known as wasted time). Names for data should be nouns, and names for functions should be verbs. Names should be as expressive as possible. Python is actually a very good example when it comes to names. Most of the time you can just guess what a function is called if you know what it does. Crazy, huh? Chapter 2 of Meaningful Names of Clean Code, Robert C. Martin, Prentice Hall is entirely dedicated to names. It's an amazing book that helped me improve my coding style in many different ways, and is a must-read if you want to take your coding to the next level.

Summary In this chapter, we've explored the built-in data types of Python. We've seen how many there are and how much can be achieved by just using them in different combinations. We've seen number types, sequences, sets, mappings, collections (and a special guest appearance by Enum), we've seen that everything is an object, we've learned the difference between mutable and immutable, and we've also learned about slicing and indexing (and, proudly, negative indexing as well). We've presented simple examples, but there's much more that you can learn about this subject, so stick your nose into the official documentation and explore. Most of all, I encourage you to try out all the exercises by yourself, get your fingers using that code, build some muscle memory, and experiment, experiment, experiment. Learn what happens when you divide by zero, when you combine different number types into a single expression, when you manage strings. Play with all data types. Exercise them, break them, discover all their methods, enjoy them, and learn them very, very well.

[ 82 ]

Built-in Data Types

Chapter 2

If your foundation is not rock solid, how good can your code be? And data is the foundation for everything. Data shapes what dances around it. The more you progress with the book, the more it's likely that you will find some discrepancies or maybe a small typo here and there in my code (or yours). You will get an error message, something will break. That's wonderful! When you code, things break all the time, you debug and fix all the time, so consider errors as useful exercises to learn something new about the language you're using, and not as failures or problems. Errors will keep coming up until your very last line of code, that's for sure, so you may as well start making your peace with them now. The next chapter is about iterating and making decisions. We'll see how actually to put those collections to use, and take decisions based on the data we're presented with. We'll start to go a little faster now that your knowledge is building up, so make sure you're comfortable with the contents of this chapter before you move to the next one. Once more, have fun, explore, break things. It's a very good way to learn.

[ 83 ]

3 Iterating and Making Decisions "Insanity: doing the same thing over and over again and expecting different results." – Albert Einstein In the previous chapter, we looked at Python's built-in data types. Now that you're familiar with data in its many forms and shapes, it's time to start looking at how a program can use it. According to Wikipedia: In computer science, control flow (or alternatively, flow of control) refers to the specification of the order in which the individual statements, instructions or function calls of an imperative program are executed or evaluated. In order to control the flow of a program, we have two main weapons: conditional programming (also known as branching) and looping. We can use them in many different combinations and variations, but in this chapter, instead of going through all the possible forms of those two constructs in a documentation fashion, I'd rather give you the basics and then I'll write a couple of small scripts with you. In the first one, we'll see how to create a rudimentary prime-number generator, while in the second one, we'll see how to apply discounts to customers based on coupons. This way, you should get a better feeling for how conditional programming and looping can be used.

Iterating and Making Decisions

Chapter 3

In this chapter, we are going to cover the following: Conditional programming Looping in Python A quick peek at the itertools module

Conditional programming Conditional programming, or branching, is something you do every day, every moment. It's about evaluating conditions: if the light is green, then I can cross; if it's raining, then I'm taking the umbrella; and if I'm late for work, then I'll call my manager. The main tool is the if statement, which comes in different forms and colors, but basically it evaluates an expression and, based on the result, chooses which part of the code to execute. As usual, let's look at an example: # conditional.1.py late = True if late: print('I need to call my manager!')

This is possibly the simplest example: when fed to the if statement, late acts as a conditional expression, which is evaluated in a Boolean context (exactly like if we were calling bool(late)). If the result of the evaluation is True, then we enter the body of the code immediately after the if statement. Notice that the print instruction is indented: this means it belongs to a scope defined by the if clause. Execution of this code yields: $ python conditional.1.py I need to call my manager!

Since late is True, the print statement was executed. Let's expand on this example: # conditional.2.py late = False if late: print('I need to call my manager!') #1 else: print('no need to call my manager...') #2

This time I set late = False, so when I execute the code, the result is different: $ python conditional.2.py no need to call my manager...

[ 85 ]

Iterating and Making Decisions

Chapter 3

Depending on the result of evaluating the late expression, we can either enter block #1 or block #2, but not both. Block #1 is executed when late evaluates to True, while block #2 is executed when late evaluates to False. Try assigning False/True values to the late name, and see how the output for this code changes accordingly. The preceding example also introduces the else clause, which becomes very handy when we want to provide an alternative set of instructions to be executed when an expression evaluates to False within an if clause. The else clause is optional, as is evident by comparing the preceding two examples.

A specialized else – elif Sometimes all you need is to do something if a condition is met (a simple if clause). At other times, you need to provide an alternative, in case the condition is False (if/else clause), but there are situations where you may have more than two paths to choose from, so, since calling the manager (or not calling them) is kind of a binary type of example (either you call or you don't), let's change the type of example and keep expanding. This time, we decide on tax percentages. If my income is less than $10,000, I won't pay any taxes. If it is between $10,000 and $30,000, I'll pay 20% in taxes. If it is between $30,000 and $100,000, I'll pay 35% in taxes, and if it's over $100,000, I'll (gladly) pay 45% in taxes. Let's put this all down into beautiful Python code: # taxes.py income = 15000 if income < 10000: tax_coefficient = elif income < 30000: tax_coefficient = elif income < 100000: tax_coefficient = else: tax_coefficient =

0.0

#1

0.2

#2

0.35

#3

0.45

#4

print('I will pay:', income * tax_coefficient, 'in taxes')

Executing the preceding code yields: $ python taxes.py I will pay: 3000.0 in taxes

[ 86 ]

Iterating and Making Decisions

Chapter 3

Let's go through the example line by line: we start by setting up the income value. In the example, my income is $15,000. We enter the if clause. Notice that this time we also introduced the elif clause, which is a contraction of else-if, and it's different from a bare else clause in that it also has its own condition. So, the if expression of income < 10000 evaluates to False, therefore block #1 is not executed. The control passes to the next condition evaluator: elif income < 30000. This one evaluates to True, therefore block #2 is executed, and because of this, Python then resumes execution after the whole if/elif/elif/else clause (which we can just call the if clause from now on). There is only one instruction after the if clause, the print call, which tells us I will pay 3000.0 in taxes this year (15,000 * 20%). Notice that the order is mandatory: if comes first, then (optionally) as many elif clauses as you need, and then (optionally) an else clause. Interesting, right? No matter how many lines of code you may have within each block, when one of the conditions evaluates to True, the associated block is executed and then execution resumes after the whole clause. If none of the conditions evaluates to True (for example, income = 200000), then the body of the else clause would be executed (block #4). This example expands our understanding of the behavior of the else clause. Its block of code is executed when none of the preceding if/elif/.../elif expressions has evaluated to True. Try to modify the value of income until you can comfortably execute all blocks at will (one per execution, of course). And then try the boundaries. This is crucial, whenever you have conditions expressed as equalities or inequalities (==, !=, , =), those numbers represent boundaries. It is essential to test boundaries thoroughly. Should I allow you to drive at 18 or 17? Am I checking your age with age < 18, or age 100 else 0 print(order_total, discount)

For simple cases like this, I find it very nice to be able to express that logic in one line instead of four. Remember, as a coder, you spend much more time reading code than writing it, so Python's conciseness is invaluable. Are you clear on how the ternary operator works? Basically, name = something if condition else something-else. So name is assigned something if condition evaluates to True, and something-else if condition evaluates to False. Now that you know everything about controlling the path of the code, let's move on to the next subject: looping.

Looping If you have any experience with looping in other programming languages, you will find Python's way of looping a bit different. First of all, what is looping? Looping means being able to repeat the execution of a code block more than once, according to the loop parameters we're given. There are different looping constructs, which serve different purposes, and Python has distilled all of them down to just two, which you can use to achieve everything you need. These are the for and while statements. While it's definitely possible to do everything you need using either of them, they serve different purposes and therefore they're usually used in different contexts. We'll explore this difference thoroughly in this chapter.

The for loop The for loop is used when looping over a sequence, such as a list, tuple, or a collection of objects. Let's start with a simple example and expand on the concept to see what the Python syntax allows us to do: # simple.for.py for number in [0, 1, 2, 3, 4]: print(number)

[ 89 ]

Iterating and Making Decisions

Chapter 3

This simple snippet of code, when executed, prints all numbers from 0 to 4. The for loop is fed the list [0, 1, 2, 3, 4] and at each iteration, number is given a value from the sequence (which is iterated sequentially, in order), then the body of the loop is executed (the print line). The number value changes at every iteration, according to which value is coming next from the sequence. When the sequence is exhausted, the for loop terminates, and the execution of the code resumes normally with the code after the loop.

Iterating over a range Sometimes we need to iterate over a range of numbers, and it would be quite unpleasant to have to do so by hardcoding the list somewhere. In such cases, the range function comes to the rescue. Let's see the equivalent of the previous snippet of code: # simple.for.py for number in range(5): print(number)

The range function is used extensively in Python programs when it comes to creating sequences: you can call it by passing one value, which acts as stop (counting from 0), or you can pass two values (start and stop), or even three (start, stop, and step). Check out the following example: >>> list(range(10)) # one value: from 0 to value (excluded) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> list(range(3, 8)) # two values: from start to stop (excluded) [3, 4, 5, 6, 7] >>> list(range(-10, 10, 4)) # three values: step is added [-10, -6, -2, 2, 6]

For the moment, ignore that we need to wrap range(...) within a list. The range object is a little bit special, but in this case, we're just interested in understanding what values it will return to us. You can see that the deal is the same with slicing: start is included, stop excluded, and optionally you can add a step parameter, which by default is 1. Try modifying the parameters of the range() call in our simple.for.py code and see what it prints. Get comfortable with it.

[ 90 ]

Iterating and Making Decisions

Chapter 3

Iterating over a sequence Now we have all the tools to iterate over a sequence, so let's build on that example: # simple.for.2.py surnames = ['Rivest', 'Shamir', 'Adleman'] for position in range(len(surnames)): print(position, surnames[position])

The preceding code adds a little bit of complexity to the game. Execution will show this result: $ 0 1 2

python simple.for.2.py Rivest Shamir Adleman

Let's use the inside-out technique to break it down, OK? We start from the innermost part of what we're trying to understand, and we expand outward. So, len(surnames) is the length of the surnames list: 3. Therefore, range(len(surnames)) is actually transformed into range(3). This gives us the range [0, 3), which is basically a sequence (0, 1, 2). This means that the for loop will run three iterations. In the first one, position will take value 0, while in the second one, it will take value 1, and finally value 2 in the third and last iteration. What is (0, 1, 2), if not the possible indexing positions for the surnames list? At position 0, we find 'Rivest', at position 1, 'Shamir', and at position 2, 'Adleman'. If you are curious about what these three men created together, change print(position, surnames[position]) to print(surnames[position][0], end=''), add a final print() outside of the loop, and run the code again. Now, this style of looping is actually much closer to languages such as Java or C++. In Python, it's quite rare to see code like this. You can just iterate over any sequence or collection, so there is no need to get the list of positions and retrieve elements out of a sequence at each iteration. It's expensive, needlessly expensive. Let's change the example into a more Pythonic form: # simple.for.3.py surnames = ['Rivest', 'Shamir', 'Adleman'] for surname in surnames: print(surname)

[ 91 ]

Iterating and Making Decisions

Chapter 3

Now that's something! It's practically English. The for loop can iterate over the surnames list, and it gives back each element in order at each interaction. Running this code will print the three surnames, one at a time. It's much easier to read, right? What if you wanted to print the position as well though? Or what if you actually needed it? Should you go back to the range(len(...)) form? No. You can use the enumerate built-in function, like this: # simple.for.4.py surnames = ['Rivest', 'Shamir', 'Adleman'] for position, surname in enumerate(surnames): print(position, surname)

This code is very interesting as well. Notice that enumerate gives back a two-tuple (position, surname) at each iteration, but still, it's much more readable (and more efficient) than the range(len(...)) example. You can call enumerate with a start parameter, such as enumerate(iterable, start), and it will start from start, rather than 0. Just another little thing that shows you how much thought has been given in designing Python so that it makes your life easier. You can use a for loop to iterate over lists, tuples, and in general anything that Python calls iterable. This is a very important concept, so let's talk about it a bit more.

Iterators and iterables According to the Python documentation (https:/​/​docs.​python.​org/​3/​glossary. html), an iterable is: An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, file objects, and objects of any classes you define with an __iter__() or __getitem__() method. Iterables can be used in a for loop and in many other places where a sequence is needed (zip(), map(), ...). When an iterable object is passed as an argument to the built-in function iter(), it returns an iterator for the object. This iterator is good for one pass over the set of values. When using iterables, it is usually not necessary to call iter() or deal with iterator objects yourself. The for statement does that automatically for you, creating a temporary unnamed variable to hold the iterator for the duration of the loop.

[ 92 ]

Iterating and Making Decisions

Chapter 3

Simply put, what happens when you write for k in sequence: ... body ..., is that the for loop asks sequence for the next element, it gets something back, it calls that something k, and then executes its body. Then, once again, the for loop asks sequence for the next element, it calls it k again, and executes the body again, and so on and so forth, until the sequence is exhausted. Empty sequences will result in zero executions of the body. Some data structures, when iterated over, produce their elements in order, such as lists, tuples, and strings, while some others don't, such as sets and dictionaries (prior to Python 3.6). Python gives us the ability to iterate over iterables, using a type of object called an iterator. According to the official documentation (https:/​/​docs.​python.​org/​3/​glossary. html), an iterator is: An object representing a stream of data. Repeated calls to the iterator's __next__() method (or passing it to the built-in function next()) return successive items in the stream. When no more data are available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its __next__() method just raise StopIteration again. Iterators are required to have an __iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted. One notable exception is code which attempts multiple iteration passes. A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop. Attempting this with an iterator will just return the same exhausted iterator object used in the previous iteration pass, making it appear like an empty container. Don't worry if you don't fully understand all the preceding legalese, you will in due time. I put it here as a handy reference for the future. In practice, the whole iterable/iterator mechanism is somewhat hidden behind the code. Unless you need to code your own iterable or iterator for some reason, you won't have to worry about this too much. But it's very important to understand how Python handles this key aspect of control flow because it will shape the way you will write your code.

[ 93 ]

Iterating and Making Decisions

Chapter 3

Iterating over multiple sequences Let's see another example of how to iterate over two sequences of the same length, in order to work on their respective elements in pairs. Say we have a list of people and a list of numbers representing the age of the people in the first list. We want to print a pair person/age on one line for all of them. Let's start with an example and let's refine it gradually: # multiple.sequences.py people = ['Conrad', 'Deepak', 'Heinrich', 'Tom'] ages = [29, 30, 34, 36] for position in range(len(people)): person = people[position] age = ages[position] print(person, age)

By now, this code should be pretty straightforward for you to understand. We need to iterate over the list of positions (0, 1, 2, 3) because we want to retrieve elements from two different lists. Executing it we get the following: $ python multiple.sequences.py Conrad 29 Deepak 30 Heinrich 34 Tom 36

This code is both inefficient and not Pythonic. It's inefficient because retrieving an element given the position can be an expensive operation, and we're doing it from scratch at each iteration. The postal worker doesn't go back to the beginning of the road each time they deliver a letter, right? They move from house to house. From one to the next one. Let's try to make it better using enumerate: # multiple.sequences.enumerate.py people = ['Conrad', 'Deepak', 'Heinrich', 'Tom'] ages = [29, 30, 34, 36] for position, person in enumerate(people): age = ages[position] print(person, age)

[ 94 ]

Iterating and Making Decisions

Chapter 3

That's better, but still not perfect. And it's still a bit ugly. We're iterating properly on people, but we're still fetching age using positional indexing, which we want to lose as well. Well, no worries, Python gives you the zip function, remember? Let's use it: # multiple.sequences.zip.py people = ['Conrad', 'Deepak', 'Heinrich', 'Tom'] ages = [29, 30, 34, 36] for person, age in zip(people, ages): print(person, age)

Ah! So much better! Once again, compare the preceding code with the first example and admire Python's elegance. The reason I wanted to show this example is twofold. On the one hand, I wanted to give you an idea of how shorter code in Python can be compared to other languages where the syntax doesn't allow you to iterate over sequences or collections as easily. And on the other hand, and much more importantly, notice that when the for loop asks zip(sequenceA, sequenceB) for the next element, it gets back a tuple, not just a single object. It gets back a tuple with as many elements as the number of sequences we feed to the zip function. Let's expand a little on the previous example in two ways, using explicit and implicit assignment: # multiple.sequences.explicit.py people = ['Conrad', 'Deepak', 'Heinrich', 'Tom'] ages = [29, 30, 34, 36] nationalities = ['Poland', 'India', 'South Africa', 'England'] for person, age, nationality in zip(people, ages, nationalities): print(person, age, nationality)

In the preceding code, we added the nationalities list. Now that we feed three sequences to the zip function, the for loop gets back a three-tuple at each iteration. Notice that the position of the elements in the tuple respects the position of the sequences in the zip call. Executing the code will yield the following result: $ python multiple.sequences.explicit.py Conrad 29 Poland Deepak 30 India Heinrich 34 South Africa Tom 36 England

[ 95 ]

Iterating and Making Decisions

Chapter 3

Sometimes, for reasons that may not be clear in a simple example such as the preceding one, you may want to explode the tuple within the body of the for loop. If that is your desire, it's perfectly possible to do so: # multiple.sequences.implicit.py people = ['Conrad', 'Deepak', 'Heinrich', 'Tom'] ages = [29, 30, 34, 36] nationalities = ['Poland', 'India', 'South Africa', 'England'] for data in zip(people, ages, nationalities): person, age, nationality = data print(person, age, nationality)

It's basically doing what the for loop does automatically for you, but in some cases you may want to do it yourself. Here, the three-tuple data that comes from zip(...) is exploded within the body of the for loop into three variables: person, age, and nationality.

The while loop In the preceding pages, we saw the for loop in action. It's incredibly useful when you need to loop over a sequence or a collection. The key point to keep in mind, when you need to be able to discriminate which looping construct to use, is that the for loop rocks when you have to iterate over a finite amount of elements. It can be a huge amount, but still, something that ends at some point. There are other cases though, when you just need to loop until some condition is satisfied, or even loop indefinitely until the application is stopped, such as cases where we don't really have something to iterate on, and therefore the for loop would be a poor choice. But fear not, for these cases, Python provides us with the while loop. The while loop is similar to the for loop, in that they both loop, and at each iteration they execute a body of instructions. What is different between them is that the while loop doesn't loop over a sequence (it can, but you have to write the logic manually and it wouldn't make any sense, you would just want to use a for loop), rather, it loops as long as a certain condition is satisfied. When the condition is no longer satisfied, the loop ends.

[ 96 ]

Iterating and Making Decisions

Chapter 3

As usual, let's see an example that will clarify everything for us. We want to print the binary representation of a positive number. In order to do so, we can use a simple algorithm that collects the remainders of division by 2 (in reverse order), and that turns out to be the binary representation of the number itself: 6 / 2 = 3 / 2 = 1 / 2 = List of Inverse

3 (remainder: 0) 1 (remainder: 1) 0 (remainder: 1) remainders: 0, 1, 1. is 1, 1, 0, which is also the binary representation of 6: 110

Let's write some code to calculate the binary representation for the number 39: 1001112: # binary.py n = 39 remainders = [] while n > 0: remainder = n % 2 # remainder of division by 2 remainders.insert(0, remainder) # we keep track of remainders n //= 2 # we divide n by 2 print(remainders)

In the preceding code, I highlighted n > 0, which is the condition to keep looping. We can make the code a little shorter (and more Pythonic), by using the divmod function, which is called with a number and a divisor, and returns a tuple with the result of the integer division and its remainder. For example, divmod(13, 5) would return (2, 3), and indeed 5 * 2 + 3 = 13: # binary.2.py n = 39 remainders = [] while n > 0: n, remainder = divmod(n, 2) remainders.insert(0, remainder) print(remainders)

In the preceding code, we have reassigned n to the result of the division by 2, and the remainder, in one single line.

[ 97 ]

Iterating and Making Decisions

Chapter 3

Notice that the condition in a while loop is a condition to continue looping. If it evaluates to True, then the body is executed and then another evaluation follows, and so on, until the condition evaluates to False. When that happens, the loop is exited immediately without executing its body. If the condition never evaluates to False, the loop becomes a socalled infinite loop. Infinite loops are used, for example, when polling from network devices: you ask the socket whether there is any data, you do something with it if there is any, then you sleep for a small amount of time, and then you ask the socket again, over and over again, without ever stopping. Having the ability to loop over a condition, or to loop indefinitely, is the reason why the for loop alone is not enough, and therefore Python provides the while loop. By the way, if you need the binary representation of a number, check out the bin function.

Just for fun, let's adapt one of the examples (multiple.sequences.py) using the while logic: # multiple.sequences.while.py people = ['Conrad', 'Deepak', 'Heinrich', 'Tom'] ages = [29, 30, 34, 36] position = 0 while position < len(people): person = people[position] age = ages[position] print(person, age) position += 1

In the preceding code, I have highlighted the initialization, condition, and update of the position variable, which makes it possible to simulate the equivalent for loop code by handling the iteration variable manually. Everything that can be done with a for loop can also be done with a while loop, even though you can see there's a bit of boilerplate you have to go through in order to achieve the same result. The opposite is also true, but unless you have a reason to do so, you ought to use the right tool for the job, and 99.9% of the time you'll be fine.

[ 98 ]

Iterating and Making Decisions

Chapter 3

So, to recap, use a for loop when you need to iterate over an iterable, and a while loop when you need to loop according to a condition being satisfied or not. If you keep in mind the difference between the two purposes, you will never choose the wrong looping construct. Let's now see how to alter the normal flow of a loop.

The break and continue statements According to the task at hand, sometimes you will need to alter the regular flow of a loop. You can either skip a single iteration (as many times as you want), or you can break out of the loop entirely. A common use case for skipping iterations is, for example, when you're iterating over a list of items and you need to work on each of them only if some condition is verified. On the other hand, if you're iterating over a collection of items, and you have found one of them that satisfies some need you have, you may decide not to continue the loop entirely and therefore break out of it. There are countless possible scenarios, so it's better to see a couple of examples. Let's say you want to apply a 20% discount to all products in a basket list for those that have an expiration date of today. The way you achieve this is to use the continue statement, which tells the looping construct (for or while) to stop execution of the body immediately and go to the next iteration, if any. This example will take us a little deeper down the rabbit hole, so be ready to jump: # discount.py from datetime import date, timedelta today = date.today() tomorrow = today + timedelta(days=1) # today + 1 day is tomorrow products = [ {'sku': '1', 'expiration_date': today, 'price': 100.0}, {'sku': '2', 'expiration_date': tomorrow, 'price': 50}, {'sku': '3', 'expiration_date': today, 'price': 20}, ] for product in products: if product['expiration_date'] != today: continue product['price'] *= 0.8 # equivalent to applying 20% discount print( 'Price for sku', product['sku'], 'is now', product['price'])

[ 99 ]

Iterating and Making Decisions

Chapter 3

We start by importing the date and timedelta objects, then we set up our products. Those with sku as 1 and 3 have an expiration date of today, which means we want to apply a 20% discount on them. We loop over each product and we inspect the expiration date. If it is not (inequality operator, !=) today, we don't want to execute the rest of the body suite, so we continue. Notice that it is not important where in the body suite you place the continue statement (you can even use it more than once). When you reach it, execution stops and goes back to the next iteration. If we run the discount.py module, this is the output: $ python discount.py Price for sku 1 is now 80.0 Price for sku 3 is now 16.0

This shows you that the last two lines of the body haven't been executed for sku number 2. Let's now see an example of breaking out of a loop. Say we want to tell whether at least one of the elements in a list evaluates to True when fed to the bool function. Given that we need to know whether there is at least one, when we find it, we don't need to keep scanning the list any further. In Python code, this translates to using the break statement. Let's write this down into code: # any.py items = [0, None, 0.0, True, 0, 7]

# True and 7 evaluate to True

found = False # this is called "flag" for item in items: print('scanning item', item) if item: found = True # we update the flag break if found: # we inspect the flag print('At least one item evaluates to True') else: print('All items evaluate to False')

[ 100 ]

Iterating and Making Decisions

Chapter 3

The preceding code is such a common pattern in programming, you will see it a lot. When you inspect items this way, basically what you do is to set up a flag variable, then start the inspection. If you find one element that matches your criteria (in this example, that evaluates to True), then you update the flag and stop iterating. After iteration, you inspect the flag and take action accordingly. Execution yields: $ python scanning scanning scanning scanning At least

any.py item 0 item None item 0.0 item True one item evaluates to True

See how execution stopped after True was found? The break statement acts exactly like the continue one, in that it stops executing the body of the loop immediately, but also, prevents any other iteration from running, effectively breaking out of the loop. The continue and break statements can be used together with no limitation in their numbers, both in the for and while looping constructs. By the way, there is no need to write code to detect whether there is at least one element in a sequence that evaluates to True. Just check out the built-in any function.

A special else clause One of the features I've seen only in the Python language is the ability to have else clauses after while and for loops. It's very rarely used, but it's definitely nice to have. In short, you can have an else suite after a for or while loop. If the loop ends normally, because of exhaustion of the iterator (for loop) or because the condition is finally not met (while loop), then the else suite (if present) is executed. In case execution is interrupted by a break statement, the else clause is not executed. Let's take an example of a for loop that iterates over a group of items, looking for one that would match some condition. In case we don't find at least one that satisfies the condition, we want to raise an exception. This means we want to arrest the regular execution of the program and signal that there was an error, or exception, that we cannot deal with. Exceptions will be the subject of Chapter 8, Testing, Profiling, and Dealing with Exceptions, so don't worry if you don't fully understand them now. Just bear in mind that they will alter the regular flow of the code.

[ 101 ]

Iterating and Making Decisions

Chapter 3

Let me now show you two examples that do exactly the same thing, but one of them is using the special for...else syntax. Say that we want to find, among a collection of people, one that could drive a car: # for.no.else.py class DriverException(Exception): pass people = [('James', 17), ('Kirk', 9), ('Lars', 13), ('Robert', 8)] driver = None for person, age in people: if age >= 18: driver = (person, age) break if driver is None: raise DriverException('Driver not found.')

Notice the flag pattern again. We set the driver to be None, then if we find one, we update the driver flag, and then, at the end of the loop, we inspect it to see whether one was found. I kind of have the feeling that those kids would drive a very metallic car, but anyway, notice that if a driver is not found, DriverException is raised, signaling to the program that execution cannot continue (we're lacking the driver). The same functionality can be rewritten a bit more elegantly using the following code: # for.else.py class DriverException(Exception): pass people = [('James', 17), ('Kirk', 9), ('Lars', 13), ('Robert', 8)] for person, age in people: if age >= 18: driver = (person, age) break else: raise DriverException('Driver not found.')

Notice that we aren't forced to use the flag pattern any more. The exception is raised as part of the for loop logic, which makes good sense because the for loop is checking on some condition. All we need is to set up a driver object in case we find one, because the rest of the code is going to use that information somewhere. Notice the code is shorter and more elegant, because the logic is now correctly grouped together where it belongs.

[ 102 ]

Iterating and Making Decisions

Chapter 3

In the Transforming Code into Beautiful, Idiomatic Python video, Raymond Hettinger suggests a much better name for the else statement associated with a for loop: nobreak. If you struggle remembering how the else works for a for loop, simply remembering this fact should help you.

Putting all this together Now that you have seen all there is to see about conditionals and loops, it's time to spice things up a little, and look at those two examples I anticipated at the beginning of this chapter. We'll mix and match here, so you can see how you can use all these concepts together. Let's start by writing some code to generate a list of prime numbers up to some limit. Please bear in mind that I'm going to write a very inefficient and rudimentary algorithm to detect primes. The important thing for you is to concentrate on those bits in the code that belong to this chapter's subject.

A prime generator According to Wikipedia: A prime number (or a prime) is a natural number greater than 1 that has no positive divisors other than 1 and itself. A natural number greater than 1 that is not a prime number is called a composite number. Based on this definition, if we consider the first 10 natural numbers, we can see that 2, 3, 5, and 7 are primes, while 1, 4, 6, 8, 9, and 10 are not. In order to have a computer tell you whether a number, N, is prime, you can divide that number by all natural numbers in the range [2, N). If any of those divisions yields zero as a remainder, then the number is not a prime. Enough chatter, let's get down to business. I'll write two versions of this, the second of which will exploit the for...else syntax: # primes.py primes = [] # this will contain the primes in the end upto = 100 # the limit, inclusive for n in range(2, upto + 1): is_prime = True # flag, new at each iteration of outer for for divisor in range(2, n): if n % divisor == 0: is_prime = False break

[ 103 ]

Iterating and Making Decisions

Chapter 3

if is_prime: # check on flag primes.append(n) print(primes)

There are a lot of things to notice in the preceding code. First of all, we set up an empty primes list, which will contain the primes at the end. The limit is 100, and you can see it's inclusive in the way we call range() in the outer loop. If we wrote range(2, upto) that would be [2, upto), right? Therefore range(2, upto + 1) gives us [2, upto + 1) == [2, upto]. So, there are two for loops. In the outer one, we loop over the candidate primes, that is, all natural numbers from 2 to upto. Inside each iteration of this outer loop, we set up a flag (which is set to True at each iteration), and then start dividing the current n by all numbers from 2 to n - 1. If we find a proper divisor for n, it means n is composite, and therefore we set the flag to False and break the loop. Notice that when we break the inner one, the outer one keeps on going normally. The reason why we break after having found a proper divisor for n is that we don't need any further information to be able to tell that n is not a prime. When we check on the is_prime flag, if it is still True, it means we couldn't find any number in [2, n) that is a proper divisor for n, therefore n is a prime. We append n to the primes list, and hop! Another iteration proceeds, until n equals 100. Running this code yields: $ python primes.py [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

Before we proceed, one question: of all the iterations of the outer loop, one of them is different from all the others. Could you tell which one, and why? Think about it for a second, go back to the code, try to figure it out for yourself, and then keep reading on. Did you figure it out? If not, don't feel bad, it's perfectly normal. I asked you to do it as a small exercise because it's what coders do all the time. The skill to understand what the code does by simply looking at it is something you build over time. It's very important, so try to exercise it whenever you can. I'll tell you the answer now: the iteration that behaves differently from all others is the first one. The reason is because in the first iteration, n is 2. Therefore the innermost for loop won't even run, because it's a for loop that iterates over range(2, 2), and what is that if not [2, 2)? Try it out for yourself, write a simple for loop with that iterable, put a print in the body suite, and see whether anything happens (it won't...).

[ 104 ]

Iterating and Making Decisions

Chapter 3

Now, from an algorithmic point of view, this code is inefficient, so let's at least make it more beautiful: # primes.else.py primes = [] upto = 100 for n in range(2, upto + 1): for divisor in range(2, n): if n % divisor == 0: break else: primes.append(n) print(primes)

Much nicer, right? The is_prime flag is gone, and we append n to the primes list when we know the inner for loop hasn't encountered any break statements. See how the code looks cleaner and reads better?

Applying discounts In this example, I want to show you a technique I like a lot. In many programming languages, other than the if/elif/else constructs, in whatever form or syntax they may come, you can find another statement, usually called switch/case, that in Python is missing. It is the equivalent of a cascade of if/elif/.../elif/else clauses, with a syntax similar to this (warning! JavaScript code!): /* switch.js */ switch (day_number) { case 1: case 2: case 3: case 4: case 5: day = "Weekday"; break; case 6: day = "Saturday"; break; case 0: day = "Sunday"; break; default: day = "";

[ 105 ]

Iterating and Making Decisions

Chapter 3

alert(day_number + ' is not a valid day number.') }

In the preceding code, we switch on a variable called day_number. This means we get its value and then we decide what case it fits in (if any). From 1 to 5 there is a cascade, which means no matter the number, [1, 5] all go down to the bit of logic that sets day as "Weekday". Then we have single cases for 0 and 6, and a default case to prevent errors, which alerts the system that day_number is not a valid day number, that is, not in [0, 6]. Python is perfectly capable of realizing such logic using if/elif/else statements: # switch.py if 1 Return a multiplied by b. __name__ -> multiplication __qualname__ -> multiplication __module__ -> __main__ __defaults__ -> (1,) __code__ -> __globals__ -> {...omitted...} __dict__ -> {} __closure__ -> None __annotations__ -> {} __kwdefaults__ -> None

I have omitted the value of the __globals__ attribute, as it was too big. An explanation of the meaning of this attribute can be found in the Callable types section of the Python Data Model documentation page (https:/​/​docs.​python.​org/​3/ reference/​datamodel.​html#the-​standard-​type-​hierarchy). Should you want to see all the attributes of an object, just call dir(object_name) and you'll be given the list of all of its attributes.

Built-in functions Python comes with a lot of built-in functions. They are available anywhere and you can get a list of them by inspecting the builtins module with dir(__builtins__), or by going to the official Python documentation. Unfortunately, I don't have the room to go through all of them here. We've already seen some of them, such as any, bin, bool, divmod, filter, float, getattr, id, int, len, list, min, print, set, tuple, type, and zip, but there are many more, which you should read at least once. Get familiar with them, experiment, write a small piece of code for each of them, and make sure you have them at your finger tips so that you can use them when you need them.

[ 141 ]

Functions, the Building Blocks of Code

Chapter 4

One final example Before we finish off this chapter, how about one last example? I was thinking we could write a function to generate a list of prime numbers up to a limit. We've already seen the code for this so let's make it a function and, to keep it interesting, let's optimize it a bit. It turns out that you don't need to divide it by all numbers from 2 to N-1 to decide whether a number, N, is prime. You can stop at √N. Moreover, you don't need to test the division for all numbers from 2 to √N, you can just use the primes in that range. I'll leave it to you to figure out why this works, if you're interested. Let's see how the code changes: # primes.py from math import sqrt, ceil def get_primes(n): """Calculate a list of primes up to n (included). """ primelist = [] for candidate in range(2, n + 1): is_prime = True root = ceil(sqrt(candidate)) # division limit for prime in primelist: # we try only the primes if prime > root: # no need to check any further break if candidate % prime == 0: is_prime = False break if is_prime: primelist.append(candidate) return primelist

The code is the same as in the previous chapter. We have changed the division algorithm so that we only test divisibility using the previously calculated primes and we stopped once the testing divisor was greater than the root of the candidate. We used the primelist result list to get the primes for the division. We calculated the root value using a fancy formula, the integer value of the ceiling of the root of the candidate. While a simple int(k ** 0.5) + 1 would have served our purpose as well, the formula I chose is cleaner and requires me to use a couple of imports, which I wanted to show you. Check out the functions in the math module, they are very interesting!

[ 142 ]

Functions, the Building Blocks of Code

Chapter 4

Documenting your code I'm a big fan of code that doesn't need documentation. When you program correctly, choose the right names and take care of the details, your code should come out as self-explanatory and documentation should not be needed. Sometimes a comment is very useful though, and so is some documentation. You can find the guidelines for documenting Python in PEP 257 - Docstring conventions (https:/​/​www.​python.​org/ dev/​peps/​pep-​0257/​), but I'll show you the basics here. Python is documented with strings, which are aptly called docstrings. Any object can be documented, and you can use either one-line or multiline docstrings. One-liners are very simple. They should not provide another signature for the function, but clearly state its purpose: # docstrings.py def square(n): """Return the square of a number n. """ return n ** 2 def get_username(userid): """Return the username of a user given their id. """ return db.get(user_id=userid).username

Using triple double-quoted strings allows you to expand easily later on. Use sentences that end in a period, and don't leave blank lines before or after. Multiline comments are structured in a similar way. There should be a one-liner that briefly gives you the gist of what the object is about, and then a more verbose description. As an example, I have documented a fictitious connect function, using the Sphinx notation, in the following example: def connect(host, port, user, password): """Connect to a database. Connect to a PostgreSQL database directly, using the given parameters. :param host: The host IP. :param port: The desired port. :param user: The connection username. :param password: The connection password. :return: The connection object. """ # body of the function here... return connection

[ 143 ]

Functions, the Building Blocks of Code

Chapter 4

Sphinx is probably the most widely used tool for creating Python documentation. In fact, the official Python documentation was written with it. It's definitely worth spending some time checking it out.

Importing objects Now that you know a lot about functions, let's look at how to use them. The whole point of writing functions is to be able to reuse them later, and in Python, this translates to importing them into the namespace where you need them. There are many different ways to import objects into a namespace, but the most common ones are import module_name and from module_name import function_name. Of course, these are quite simplistic examples, but bear with me for the time being. The import module_name form finds the module_name module and defines a name for it in the local namespace where the import statement is executed. The from module_name import identifier form is a little bit more complicated than that, but basically does the same thing. It finds module_name and searches for an attribute (or a submodule) and stores a reference to identifier in the local namespace. Both forms have the option to change the name of the imported object using the as clause: from mymodule import myfunc as better_named_func

Just to give you a flavor of what importing looks like, here's an example from a test module of one of my projects (notice that the blank lines between blocks of imports follow the guidelines from PEP 8 at https:/​/​www.​python.​org/​dev/​peps/​pep-​0008/ #imports: standard library, third party, and local code): from datetime import datetime, timezone # two imports on the same line from unittest.mock import patch # single import import pytest

# third party library

from core.models import ( Exam, Exercise, Solution, )

# multiline import

[ 144 ]

Functions, the Building Blocks of Code

Chapter 4

When you have a structure of files starting in the root of your project, you can use the dot notation to get to the object you want to import into your current namespace, be it a package, a module, a class, a function, or anything else. The from module import syntax also allows a catch-all clause, from module import *, which is sometimes used to get all the names from a module into the current namespace at once, but it's frowned upon for several reasons, such as performance and the risk of silently shadowing other names. You can read all that there is to know about imports in the official Python documentation but, before we leave the subject, let me give you a better example. Imagine that you have defined a couple of functions: square(n) and cube(n) in a module, funcdef.py, which is in the lib folder. You want to use them in a couple of modules that are at the same level of the lib folder, called func_import.py and func_from.py. Showing the tree structure of that project produces something like this: ├── func_from.py ├── func_import.py ├── lib ├── funcdef.py └── __init__.py

Before I show you the code of each module, please remember that in order to tell Python that it is actually a package, we need to put a __init__.py module in it. There are two things to note about the __init__.py file. First of all, it is a fully-fledged Python module so you can put code into it as you would with any other module. Second, as of Python 3.3, its presence is no longer required to make a folder be interpreted as a Python package. The code is as follows: # funcdef.py def square(n): return n ** 2 def cube(n): return n ** 3 # func_import.py import lib.funcdef print(lib.funcdef.square(10)) print(lib.funcdef.cube(10)) # func_from.py

[ 145 ]

Functions, the Building Blocks of Code

Chapter 4

from lib.funcdef import square, cube print(square(10)) print(cube(10))

Both these files, when executed, print 100 and 1000. You can see how differently we then access the square and cube functions, according to how and what we imported in the current scope.

Relative imports The imports we've seen so far are called absolute, that is, they define the whole path of the module that we want to import, or from which we want to import an object. There is another way of importing objects into Python, which is called a relative import. It's helpful in situations where we want to rearrange the structure of large packages without having to edit sub-packages, or when we want to make a module inside a package able to import itself. Relative imports are done by adding as many leading dots in front of the module as the number of folders we need to backtrack, in order to find what we're searching for. Simply put, it is something such as this: from .mymodule import myfunc

For a complete explanation of relative imports, refer to PEP 328 (https:/​/​www. python.​org/​dev/​peps/​pep-​0328/​). In later chapters, we'll create projects using different libraries and we'll use several different types of imports, including relative ones, so make sure you take a bit of time to read up about it in the official Python documentation.

Summary In this chapter, we explored the world of functions. They are extremely important and, from now on, we'll use them basically everywhere. We talked about the main reasons for using them, the most important of which are code reuse and implementation hiding.

[ 146 ]

Functions, the Building Blocks of Code

Chapter 4

We saw that a function object is like a box that takes optional inputs and produces outputs. We can feed input values to a function in many different ways, using positional and keyword arguments, and using variable syntax for both types. Now you should know how to write a function, document it, import it into your code, and call it. The next chapter will force me to push my foot down on the throttle even more, so I suggest you take any opportunity you get to consolidate and enrich the knowledge you've gathered so far by putting your nose into the Python official documentation.

[ 147 ]

5 Saving Time and Memory "It's not the daily increase but daily decrease. Hack away at the unessential." – Bruce Lee I love this quote from Bruce Lee. He was such a wise man! Especially, the second part, "hack away at the unessential", is to me what makes a computer program elegant. After all, if there is a better way of doing things so that we don't waste time or memory, why not? Sometimes, there are valid reasons for not pushing our code up to the maximum limit: for example, sometimes to achieve a negligible improvement, we have to sacrifice on readability or maintainability. Does it make any sense to have a web page served in 1 second with unreadable, complicated code, when we can serve it in 1.05 seconds with readable, clean code? No, it makes no sense. On the other hand, sometimes it's perfectly reasonable to try to shave off a millisecond from a function, especially when the function is meant to be called thousands of times. Every millisecond you save there means one second saved per thousands of calls, and this could be meaningful for your application. In light of these considerations, the focus of this chapter will not be to give you the tools to push your code to the absolute limits of performance and optimization "no matter what," but rather, to enable you to write efficient, elegant code that reads well, runs fast, and doesn't waste resources in an obvious way. In this chapter, we are going to cover the following: The map, zip, and filter functions Comprehensions Generators

Saving Time and Memory

Chapter 5

I will perform several measurements and comparisons, and cautiously draw some conclusions. Please do keep in mind that on a different box with a different setup or a different operating system, results may vary. Take a look at this code: # squares.py def square1(n): return n ** 2 def square2(n): return n * n

# squaring through the power operator

# squaring through multiplication

Both functions return the square of n, but which is faster? From a simple benchmark I ran on them, it looks like the second is slightly faster. If you think about it, it makes sense: calculating the power of a number involves multiplication and therefore, whatever algorithm you may use to perform the power operation, it's not likely to beat a simple multiplication such as the one in square2. Do we care about this result? In most cases, no. If you're coding an e-commerce website, chances are you won't ever even need to raise a number to the second power, and if you do, it's likely to be a sporadic operation. You don't need to concern yourself with saving a fraction of a microsecond on a function you call a few times. So, when does optimization become important? One very common case is when you have to deal with huge collections of data. If you're applying the same function on a million customer objects, then you want your function to be tuned up to its best. Gaining 1/10 of a second on a function called one million times saves you 100,000 seconds, which is about 27.7 hours. That's not the same, right? So, let's focus on collections, and let's see which tools Python gives you to handle them with efficiency and grace. Many of the concepts we will see in this chapter are based on those of the iterator and iterable. Simply put, the ability for an object to return its next element when asked, and to raise a StopIteration exception when exhausted. We'll see how to code a custom iterator and iterable objects in Chapter 6, OOP, Decorators, and Iterators. Due to the nature of the objects we're going to explore in this chapter, I was often forced to wrap the code in a list constructor. This is because passing an iterator/generator to list(...) exhausts it and puts all the generated items in a newly created list, which I can easily print to show you its content. This technique hinders readability, so let me introduce an alias for list: # alias.py >>> range(7)

[ 149 ]

Saving Time and Memory

Chapter 5

range(0, 7) >>> list(range(7)) # put all elements in a list to view them [0, 1, 2, 3, 4, 5, 6] >>> _ = list # create an "alias" to list >>> _(range(7)) # same as list(range(7)) [0, 1, 2, 3, 4, 5, 6]

Of the three sections I have highlighted, the first one is the call we need to do in order to show what would be generated by range(7), the second one is the moment when I create the alias to list (I chose the hopefully unobtrusive underscore), and the third one is the equivalent call, when I use the alias instead of list. Hopefully readability will benefit from this, and please keep in mind that I will assume this alias to have been defined for all the code in this chapter.

The map, zip, and filter functions We'll start by reviewing map, filter, and zip, which are the main built-in functions one can employ when handling collections, and then we'll learn how to achieve the same results using two very important constructs: comprehensions and generators. Fasten your seatbelt!

map According to the official Python documentation: map(function, iterable, ...) returns an iterator that applies function to every item of iterable, yielding the results. If additional iterable arguments are passed, function must take that many arguments and is applied to the items from all iterables in parallel. With multiple iterables, the iterator stops when the shortest iterable is exhausted. We will explain the concept of yielding later on in the chapter. For now, let's translate this into code—we'll use a lambda function that takes a variable number of positional arguments, and just returns them as a tuple: # map.example.py >>> map(lambda *a: a, range(3)) # 1 iterable # Not useful! Let's use alias >>> _(map(lambda *a: a, range(3))) # 1 iterable

[ 150 ]

Saving Time and Memory

Chapter 5

[(0,), (1,), (2,)] >>> _(map(lambda *a: a, range(3), 'abc')) # 2 iterables [(0, 'a'), (1, 'b'), (2, 'c')] >>> _(map(lambda *a: a, range(3), 'abc', range(4, 7))) # 3 [(0, 'a', 4), (1, 'b', 5), (2, 'c', 6)] >>> # map stops at the shortest iterator >>> _(map(lambda *a: a, (), 'abc')) # empty tuple is shortest [] >>> _(map(lambda *a: a, (1, 2), 'abc')) # (1, 2) shortest [(1, 'a'), (2, 'b')] >>> _(map(lambda *a: a, (1, 2, 3, 4), 'abc')) # 'abc' shortest [(1, 'a'), (2, 'b'), (3, 'c')]

In the preceding code, you can see why we have to wrap calls in list(...) (or its alias, _, in this case). Without it, I get the string representation of a map object, which is not really useful in this context, is it? You can also notice how the elements of each iterable are applied to the function; at first, the first element of each iterable, then the second one of each iterable, and so on. Notice also that map stops when the shortest of the iterables we called it with is exhausted. This is actually a very nice behavior; it doesn't force us to level off all the iterables to a common length, and it doesn't break if they aren't all the same length. map is very useful when you have to apply the same function to one or more

collections of objects. As a more interesting example, let's see the decorate-sortundecorate idiom (also known as Schwartzian transform). It's a technique that was extremely popular when Python sorting wasn't providing key-functions, and therefore is less used today, but it's a cool trick that still comes in handy once in a while. Let's see a variation of it in the next example: we want to sort in descending order by the sum of credits accumulated by students, so to have the best student at position 0. We write a function to produce a decorated object, we sort, and then we undecorate. Each student has credits in three (possibly different) subjects. In this context, to decorate an object means to transform it, either adding extra data to it, or putting it into another object, in a way that allows us to be able to sort the original objects the way we want. This technique has nothing to do with Python decorators, which we will explore later on in the book. After the sorting, we revert the decorated objects to get the original ones from them. This is called to undecorate: # decorate.sort.undecorate.py students = [ dict(id=0, credits=dict(math=9, physics=6, history=7)), dict(id=1, credits=dict(math=6, physics=7, latin=10)),

[ 151 ]

Saving Time and Memory

Chapter 5

dict(id=2, credits=dict(history=8, physics=9, chemistry=10)), dict(id=3, credits=dict(math=5, physics=5, geography=7)), ] def decorate(student): # create a 2-tuple (sum of credits, student) from student dict return (sum(student['credits'].values()), student) def undecorate(decorated_student): # discard sum of credits, return original student dict return decorated_student[1] students = sorted(map(decorate, students), reverse=True) students = _(map(undecorate, students))

Let's start by understanding what each student object is. In fact, let's print the first one: {'credits': {'history': 7, 'math': 9, 'physics': 6}, 'id': 0}

You can see that it's a dictionary with two keys: id and credits. The value of credits is also a dictionary in which there are three subject/grade key/value pairs. As I'm sure you recall from our visit in the data structures world, calling dict.values() returns an object similar to iterable, with only the values. Therefore, sum(student['credits'].values()) for the first student is equivalent to sum((9, 6, 7)). Let's print the result of calling decorate with the first student: >>> decorate(students[0]) (22, {'credits': {'history': 7, 'math': 9, 'physics': 6}, 'id': 0})

If we decorate all the students like this, we can sort them on their total amount of credits by just sorting the list of tuples. In order to apply the decoration to each item in students, we call map(decorate, students). Then we sort the result, and then we undecorate in a similar fashion. If you have gone through the previous chapters correctly, understanding this code shouldn't be too hard. Printing students after running the whole code yields: $ python decorate.sort.undecorate.py [{'credits': {'chemistry': 10, 'history': 8, 'physics': 9}, 'id': 2}, {'credits': {'latin': 10, 'math': 6, 'physics': 7}, 'id': 1}, {'credits': {'history': 7, 'math': 9, 'physics': 6}, 'id': 0}, {'credits': {'geography': 7, 'math': 5, 'physics': 5}, 'id': 3}]

[ 152 ]

Saving Time and Memory

Chapter 5

And you can see, by the order of the student objects, that they have indeed been sorted by the sum of their credits. For more on the decorate-sort-undecorate idiom, there's a very nice introduction in the sorting how-to section of the official Python documentation (https:/​/​docs.​python.​org/​3.​7/​howto/​sorting. html#the-​old-​way-​using-​decorate-​sort-​undecorate). One thing to notice about the sorting part: what if two or more students share the same total sum? The sorting algorithm would then proceed to sort the tuples by comparing the student objects with each other. This doesn't make any sense, and in more complex cases, could lead to unpredictable results, or even errors. If you want to be sure to avoid this issue, one simple solution is to create a three-tuple instead of a two-tuple, having the sum of credits in the first position, the position of the student object in the students list in the second one, and the student object itself in the third one. This way, if the sum of credits is the same, the tuples will be sorted against the position, which will always be different and therefore enough to resolve the sorting between any pair of tuples.

zip We've already covered zip in the previous chapters, so let's just define it properly and then I want to show you how you could combine it with map. According to the Python documentation: zip(*iterables) returns an iterator of tuples, where the i-th tuple contains the i-th element from each of the argument sequences or iterables. The iterator stops when the shortest input iterable is exhausted. With a single iterable argument, it returns an iterator of 1-tuples. With no arguments, it returns an empty iterator. Let's see an example: # zip.grades.py >>> grades = [18, 23, 30, 27] >>> avgs = [22, 21, 29, 24] >>> _(zip(avgs, grades)) [(22, 18), (21, 23), (29, 30), (24, 27)] >>> _(map(lambda *a: a, avgs, grades)) # equivalent to zip [(22, 18), (21, 23), (29, 30), (24, 27)]

[ 153 ]

Saving Time and Memory

Chapter 5

In the preceding code, we're zipping together the average and the grade for the last exam, for each student. Notice how easy it is to reproduce zip using map (last two instructions of the example). Here as well, to visualize results we have to use our _ alias. A simple example on the combined use of map and zip could be a way of calculating the element-wise maximum amongst sequences, that is, the maximum of the first element of each sequence, then the maximum of the second one, and so on: # maxims.py >>> a = [5, 9, 2, 4, 7] >>> b = [3, 7, 1, 9, 2] >>> c = [6, 8, 0, 5, 3] >>> maxs = map(lambda n: max(*n), zip(a, b, c)) >>> _(maxs) [6, 9, 2, 9, 7]

Notice how easy it is to calculate the max values of three sequences. zip is not strictly needed of course, we could just use map. Sometimes it's hard, when showing a simple example, to grasp why using a technique might be good or bad. We forget that we aren't always in control of the source code, we might have to use a third-party library, which we can't change the way we want. Having different ways to work with data is therefore really helpful.

filter According to the Python documentation: filter(function, iterable) construct an iterator from those elements of iterable for which function returns True. iterable may be either a sequence, a container which supports iteration, or an iterator. If function is None, the identity function is assumed, that is, all elements of iterable that are false are removed. Let's see a very quick example: # filter.py >>> test = [2, 5, 8, 0, 0, 1, 0] >>> _(filter(None, test)) [2, 5, 8, 1] >>> _(filter(lambda x: x, test)) # equivalent to previous one [2, 5, 8, 1] >>> _(filter(lambda x: x > 4, test)) # keep only items > 4 [5, 8]

[ 154 ]

Saving Time and Memory

Chapter 5

In the preceding code, notice how the second call to filter is equivalent to the first one. If we pass a function that takes one argument and returns the argument itself, only those arguments that are True will make the function return True, therefore this behavior is exactly the same as passing None. It's often a very good exercise to mimic some of the built-in Python behaviors. When you succeed, you can say you fully understand how Python behaves in a specific situation. Armed with map, zip, and filter (and several other functions from the Python standard library) we can massage sequences very effectively. But those functions are not the only way to do it. So let's see one of the nicest features of Python: comprehensions.

Comprehensions Comprehensions are a concise notation, both perform some operation for a collection of elements, and/or select a subset of them that meet some condition. They are borrowed from the functional programming language Haskell (https:/​/​www. haskell.​org/​), and contribute to giving Python a functional flavor, together with iterators and generators. Python offers you different types of comprehensions: list, dict, and set. We'll concentrate on the first one for now, and then it will be easy to explain the other two. Let's start with a very simple example. I want to calculate a list with the squares of the first 10 natural numbers. How would you do it? There are a couple of equivalent ways: # squares.map.py # If you code like this you are not a Python dev! ;) >>> squares = [] >>> for n in range(10): ... squares.append(n ** 2) ... >>> squares [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] # This is better, one line, nice and readable >>> squares = map(lambda n: n**2, range(10)) >>> _(squares) [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

[ 155 ]

Saving Time and Memory

Chapter 5

The preceding example should be nothing new for you. Let's see how to achieve the same result using a list comprehension: # squares.comprehension.py >>> [n ** 2 for n in range(10)] [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

As simple as that. Isn't it elegant? Basically we have put a for loop within square brackets. Let's now filter out the odd squares. I'll show you how to do it with map and filter first, and then using a list comprehension again: # even.squares.py # using map and filter sq1 = list( map(lambda n: n ** 2, filter(lambda n: not n % 2, range(10))) ) # equivalent, but using list comprehensions sq2 = [n ** 2 for n in range(10) if not n % 2] print(sq1, sq1 == sq2)

# prints: [0, 4, 16, 36, 64] True

I think that now the difference in readability is evident. The list comprehension reads much better. It's almost English: give me all squares (n ** 2) for n between 0 and 9 if n is even. According to the Python documentation: A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses. The result will be a new list resulting from evaluating the expression in the context of the for and if clauses which follow it.

Nested comprehensions Let's see an example of nested loops. It's very common when dealing with algorithms to have to iterate on a sequence using two placeholders. The first one runs through the whole sequence, left to right. The second one as well, but it starts from the first one, instead of 0. The concept is that of testing all pairs without duplication. Let's see the classical for loop equivalent: # pairs.for.loop.py items = 'ABCD' pairs = []

[ 156 ]

Saving Time and Memory

Chapter 5

for a in range(len(items)): for b in range(a, len(items)): pairs.append((items[a], items[b]))

If you print pairs at the end, you get: $ python pairs.for.loop.py [('A', 'A'), ('A', 'B'), ('A', 'C'), ('A', 'D'), ('B', 'B'), ('B', 'C'), ('B', 'D'), ('C', 'C'), ('C', 'D'), ('D', 'D')]

All the tuples with the same letter are those where b is at the same position as a. Now, let's see how we can translate this in a list comprehension: # pairs.list.comprehension.py items = 'ABCD' pairs = [(items[a], items[b]) for a in range(len(items)) for b in range(a, len(items))]

This version is just two lines long and achieves the same result. Notice that in this particular case, because the for loop over b has a dependency on a, it must follow the for loop over a in the comprehension. If you swap them around, you'll get a name error.

Filtering a comprehension We can apply filtering to a comprehension. Let's do it first with filter. Let's find all Pythagorean triples whose short sides are numbers smaller than 10. We obviously don't want to test a combination twice, and therefore we'll use a trick similar to the one we saw in the previous example: # pythagorean.triple.py from math import sqrt # this will generate all possible pairs mx = 10 triples = [(a, b, sqrt(a**2 + b**2)) for a in range(1, mx) for b in range(a, mx)] # this will filter out all non pythagorean triples triples = list( filter(lambda triple: triple[2].is_integer(), triples)) print(triples)

# prints: [(3, 4, 5.0), (6, 8, 10.0)]

[ 157 ]

Saving Time and Memory

Chapter 5

A Pythagorean triple is a triple (a, b, c) of integer numbers satisfying the equation a2 + b2 = c2.

In the preceding code, we generated a list of three-tuples, triples. Each tuple contains two integer numbers (the legs), and the hypotenuse of the Pythagorean triangle whose legs are the first two numbers in the tuple. For example, when a is 3 and b is 4, the tuple will be (3, 4, 5.0), and when a is 5 and b is 7, the tuple will be (5, 7, 8.602325267042627). After having all the triples done, we need to filter out all those that don't have a hypotenuse that is an integer number. In order to do this, we filter based on float_number.is_integer() being True. This means that of the two example tuples I showed you before, the one with 5.0 hypotenuse will be retained, while the one with the 8.602325267042627 hypotenuse will be discarded. This is good, but I don't like that the triple has two integer numbers and a float. They are supposed to be all integers, so let's use map to fix this: # pythagorean.triple.int.py from math import sqrt mx = 10 triples = [(a, b, sqrt(a**2 + b**2)) for a in range(1, mx) for b in range(a, mx)] triples = filter(lambda triple: triple[2].is_integer(), triples) # this will make the third number in the tuples integer triples = list( map(lambda triple: triple[:2] + (int(triple[2]), ), triples)) print(triples)

# prints: [(3, 4, 5), (6, 8, 10)]

Notice the step we added. We take each element in triples and we slice it, taking only the first two elements in it. Then, we concatenate the slice with a one-tuple, in which we put the integer version of that float number that we didn't like. Seems like a lot of work, right? Indeed it is. Let's see how to do all this with a list comprehension: # pythagorean.triple.comprehension.py from math import sqrt # this step is the same as before mx = 10 triples = [(a, b, sqrt(a**2 + b**2)) for a in range(1, mx) for b in range(a, mx)] # here we combine filter and map in one CLEAN list comprehension

[ 158 ]

Saving Time and Memory

Chapter 5

triples = [(a, b, int(c)) for a, b, c in triples if c.is_integer()] print(triples) # prints: [(3, 4, 5), (6, 8, 10)]

I know. It's much better, isn't it? It's clean, readable, shorter. In other words, it's elegant. I'm going quite fast here, as anticipated in the Summary of Chapter 4, Functions, the Building Blocks of Code. Are you playing with this code? If not, I suggest you do. It's very important that you play around, break things, change things, see what happens. Make sure you have a clear understanding of what is going on. You want to become a ninja, right?

dict comprehensions Dictionary and set comprehensions work exactly like the list ones, only there is a little difference in the syntax. The following example will suffice to explain everything you need to know: # dictionary.comprehensions.py from string import ascii_lowercase lettermap = dict((c, k) for k, c in enumerate(ascii_lowercase, 1))

If you print lettermap, you will see the following (I omitted the middle results, you get the gist): $ python dictionary.comprehensions.py {'a': 1, 'b': 2, ... 'y': 25, 'z': 26}

What happens in the preceding code is that we're feeding the dict constructor with a comprehension (technically, a generator expression, we'll see it in a bit). We tell the dict constructor to make key/value pairs from each tuple in the comprehension. We enumerate the sequence of all lowercase ASCII letters, starting from 1, using enumerate. Piece of cake. There is also another way to do the same thing, which is closer to the other dictionary syntax: lettermap = {c: k for k, c in enumerate(ascii_lowercase, 1)}

It does exactly the same thing, with a slightly different syntax that highlights a bit more of the key: value part.

[ 159 ]

Saving Time and Memory

Chapter 5

Dictionaries do not allow duplication in the keys, as shown in the following example: # dictionary.comprehensions.duplicates.py word = 'Hello' swaps = {c: c.swapcase() for c in word} print(swaps) # prints: {'H': 'h', 'e': 'E', 'l': 'L', 'o': 'O'}

We create a dictionary with keys, the letters in the 'Hello' string, and values of the same letters, but with the case swapped. Notice there is only one 'l': 'L' pair. The constructor doesn't complain, it simply reassigns duplicates to the latest value. Let's make this clearer with another example; let's assign to each key its position in the string: # dictionary.comprehensions.positions.py word = 'Hello' positions = {c: k for k, c in enumerate(word)} print(positions) # prints: {'H': 0, 'e': 1, 'l': 3, 'o': 4}

Notice the value associated with the letter 'l': 3. The 'l': 2 pair isn't there; it has been overridden by 'l': 3.

set comprehensions The set comprehensions are very similar to list and dictionary ones. Python allows both the set() constructor to be used, or the explicit {} syntax. Let's see one quick example: # set.comprehensions.py word = 'Hello' letters1 = set(c for c in word) letters2 = {c for c in word} print(letters1) # prints: {'H', 'o', 'e', 'l'} print(letters1 == letters2) # prints: True

Notice how for set comprehensions, as for dictionaries, duplication is not allowed and therefore the resulting set has only four letters. Also, notice that the expressions assigned to letters1 and letters2 produce equivalent sets. The syntax used to create letters2 is very similar to the one we can use to create a dictionary comprehension. You can spot the difference only by the fact that dictionaries require keys and values, separated by columns, while sets don't.

[ 160 ]

Saving Time and Memory

Chapter 5

Generators Generators are very powerful tool that Python gifts us with. They are based on the concepts of iteration, as we said before, and they allow for coding patterns that combine elegance with efficiency. Generators are of two types: Generator functions: These are very similar to regular functions, but instead of returning results through return statements, they use yield, which allows them to suspend and resume their state between each call Generator expressions: These are very similar to the list comprehensions we've seen in this chapter, but instead of returning a list they return an object that produces results one by one

Generator functions Generator functions behave like regular functions in all respects, except for one difference. Instead of collecting results and returning them at once, they are automatically turned into iterators that yield results one at a time when you call next on them. Generator functions are automatically turned into their own iterators by Python. This is all very theoretical so, let's make it clear why such a mechanism is so powerful, and then let's see an example. Say I asked you to count out loud from 1 to 1,000,000. You start, and at some point I ask you to stop. After some time, I ask you to resume. At this point, what is the minimum information you need to be able to resume correctly? Well, you need to remember the last number you called. If I stopped you after 31,415, you will just go on with 31,416, and so on. The point is, you don't need to remember all the numbers you said before 31,415, nor do you need them to be written down somewhere. Well, you may not know it, but you're behaving like a generator already! Take a good look at the following code: # first.n.squares.py def get_squares(n): # classic function approach return [x ** 2 for x in range(n)] print(get_squares(10))

[ 161 ]

Saving Time and Memory

Chapter 5

def get_squares_gen(n): # generator approach for x in range(n): yield x ** 2 # we yield, we don't return print(list(get_squares_gen(10)))

The result of the two print statements will be the same: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]. But there is a huge difference between the two functions. get_squares is a classic function that collects all the squares of numbers in [0, n) in a list, and returns it. On the other hand, get_squares_gen is a generator, and behaves very differently. Each time the interpreter reaches the yield line, its execution is suspended. The only reason those print statements return the same result is because we fed get_squares_gen to the list constructor, which exhausts the generator completely by asking the next element until a StopIteration is raised. Let's see this in detail: # first.n.squares.manual.py def get_squares_gen(n): for x in range(n): yield x ** 2 squares = get_squares_gen(4) # this creates a generator object print(squares) # print(next(squares)) # prints: 0 print(next(squares)) # prints: 1 print(next(squares)) # prints: 4 print(next(squares)) # prints: 9 # the following raises StopIteration, the generator is exhausted, # any further call to next will keep raising StopIteration print(next(squares))

In the preceding code, each time we call next on the generator object, we either start it (first next) or make it resume from the last suspension point (any other next). The first time we call next on it, we get 0, which is the square of 0, then 1, then 4, then 9, and since the for loop stops after that (n is 4), then the generator naturally ends. A classic function would at that point just return None, but in order to comply with the iteration protocol, a generator will instead raise a StopIteration exception. This explains how a for loop works. When you call for k in range(n), what happens under the hood is that the for loop gets an iterator out of range(n) and starts calling next on it, until StopIteration is raised, which tells the for loop that the iteration has reached its end.

[ 162 ]

Saving Time and Memory

Chapter 5

Having this behavior built into every iteration aspect of Python makes generators even more powerful because once we write them, we'll be able to plug them into whatever iteration mechanism we want. At this point, you're probably asking yourself why you would want to use a generator instead of a regular function. Well, the title of this chapter should suggest the answer. I'll talk about performances later, so for now let's concentrate on another aspect: sometimes generators allow you to do something that wouldn't be possible with a simple list. For example, say you want to analyze all permutations of a sequence. If the sequence has a length of N, then the number of its permutations is N!. This means that if the sequence is 10 elements long, the number of permutations is 3,628,800. But a sequence of 20 elements would have 2,432,902,008,176,640,000 permutations. They grow factorially. Now imagine you have a classic function that is attempting to calculate all permutations, put them in a list, and return it to you. With 10 elements, it would require probably a few dozen seconds, but for 20 elements there is simply no way that it can be done. On the other hand, a generator function will be able to start the computation and give you back the first permutation, then the second, and so on. Of course you won't have the time to parse them all, there are too many, but at least you'll be able to work with some of them. Remember when we were talking about the break statement in for loops? When we found a number dividing a candidate prime we were breaking the loop, and there was no need to go on. Sometimes it's exactly the same, only the amount of data you have to iterate over is so huge that you cannot keep it all in memory in a list. In this case, generators are invaluable: they make possible what wouldn't be possible otherwise. So, in order to save memory (and time), use generator functions whenever possible. It's also worth noting that you can use the return statement in a generator function. It will produce a StopIteration exception to be raised, effectively ending the iteration. This is extremely important. If a return statement were actually to make the function return something, it would break the iteration protocol. Python's consistency prevents this, and allows us great ease when coding. Let's see a quick example: # gen.yield.return.py def geometric_progression(a, q): k = 0

[ 163 ]

Saving Time and Memory

Chapter 5

while True: result = a * q**k if result n rule. It's worth noting how I calculated sqrt(N), that is, N**.5, which is just another way to do it that I wanted to show you. At #6, you can see the filtering conditions to make the triples primitive: (m - n) % 2 evaluates to True when (m - n) is odd, and gcd(m, n) == 1 means m and n are coprime. With these in place, we know the triples will be primitive. This takes care of the innermost generator expression. The outermost one starts at #2, and finishes at #7. We take the triples (a, b, c) in (...innermost generator...) such that c >> "{title}".format(title="SuperBook") 'SuperBook'

The syntax equivalent in a Django template is {{ title }}. Rendering with the same context will produce the same output as follows: >>> from django.template import Template, Context >>> Template("{{ title }}").render(Context({"title": "SuperBook"})) 'SuperBook'

Attributes Dot is a multipurpose operator in Django templates. There are three different kinds of operations: attribute lookup, dictionary lookup, or list-index lookup (in that order). In Python, first, let's define the context variables and classes: >>> class DrOct: arms = 4 def speak(self): return "You have a train to catch." >>> mydict = {"key":"value"} >>> mylist = [10, 20, 30]

Let's take a look at Python's syntax for the three kinds of lookups: >>> "Dr. Oct has {0} arms and says: {1}".format(DrOct().arms, DrOct().speak()) 'Dr. Oct has 4 arms and says: You have a train to catch.' >>> mydict["key"] 'value' >>> mylist[1] 20

[ 617 ]

Templates

Chapter 21

In Django's template equivalent, it is as follows: Dr. Oct has {{ s.arms }} arms and says: {{ s.speak }} {{ mydict.key }} {{ mylist.1 }}

Notice how speak, a method that takes no arguments except self, is treated like an attribute here.

Filters Sometimes, variables need to be modified. Essentially, you would like to call functions on these variables. Instead of chaining function calls, such as var.method1().method2(arg), Django uses the pipe syntax {{ var|method1|method2:"arg" }}, which is similar to Unix filters. However, this syntax only works for built-in or custom-defined filters. Another limitation is that filters cannot access the template context. They only work with the data passed into them and their arguments. Hence, they are primarily used to alter the variables in the template context. Run the following command in Python: >>> title="SuperBook" >>> title.upper()[:5] 'SUPER'

The following is its Django template equivalent: {{ title|upper|slice:':5' }}"

Tags Programming languages can do more than just display variables. Django's template language has many familiar syntactic forms, such as if and for. They should be written in the tag syntax such as {% if %}. Several template-specific forms, such as include and block, are also written in the tag syntax.

[ 618 ]

Templates

Chapter 21

In Python shell: >>> if 1==1: ... print(" Date is {0} ".format(time.strftime("%d-%m-%Y"))) Date is 30-05-2018

The following is its corresponding Django template form: {% if 1 == 1 %} Date is {% now 'd-m-Y' %} {% endif %}

Philosophy – don't invent a programming language A common question among beginners is how to perform numeric computations such as finding percentages in templates. As a design philosophy, the template system does not intentionally allow the following: Assignment to variables Function call arguments Advanced logic This decision was made to prevent you from adding business logic in templates. From my experience with PHP or ASP-like languages, mixing logic with presentation can be a maintenance nightmare. However, you can write custom template tags (which will be covered shortly) to perform any computation, especially if it is presentation-related. Best Practice Keep business logic out of your templates. Despite this advice, some prefer a slightly more powerful templating engine. In which case, Jinja2 might be what you need.

[ 619 ]

Templates

Chapter 21

Jinja2 Jinja2 is very similar to DTL in syntax. But it has a slightly different philosophy in certain places. For instance, in DTL the method call is implied as in the following example: {% for post in user.public_posts %} ... {% endfor %}

But in Jinja2, we invoke the public_posts method similar to a Python function call: {% for post in user.public_posts() %} ... {% endfor %}

This means that in Jinja2 you can call functions with arguments, unlike DTL. Refer to the Jinja2 documentation for more such subtle differences. Jinja2 is usually chosen for the following reasons: Familiarity: If your template designers are already comfortable using Jinja2 Whitespace control: Jinja2 has finer control over whitespace after the tags get rendered Customizability: Most aspects of Jinja2, from string defining markup to extensions, can be easily configured Performance: Some benchmarks show Jinja2 is faster than Django Autoescape: By default, Jinja2 disables XML/HTML autoescaping for performance In most cases, none of these advantages are overwhelming enough to use Jinja2. This also goes for using other templating engines such as Mako or Genshi. The familiarity of using DTL reduces the learning curve to anyone new to your project. It is also well integrated and tested. Finally, you might have to replicate Django-specific template tags such as static or url. Unless you have a very good reason not to, I would advise sticking to Django's own template language. The rest of this chapter would be using DTL.

[ 620 ]

Templates

Chapter 21

Organizing templates The default project layout created by the startproject command does not define a location for your templates. This is very easy to configure. Create a directory named templates in your project's root directory. Specify the value for DIRS inside the TEMPLATES variable in your settings.py: (can be found within superbook/settings/base.py in our superbook project) BASE_DIR = os.path.dirname(os.path.dirname(__file__)) TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [os.path.join(BASE_DIR, 'templates')], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ]

That's all. For example, you can add a template called about.html and refer to it in the urls.py file as follows: urlpatterns = [ path('about/', TemplateView.as_view(template_name='about.html'), name='about'),

Your templates can also reside within your apps (if APP_DIRS is true). Creating a templates directory inside your app directory is ideal to store your app-specific templates. Here are some good practices to organize your templates: Keep all app-specific templates inside the app's template directory within a separate directory, for example projroot/app/templates/app/template.html— notice how app appears twice in the path Use the .html extension for your templates

[ 621 ]

Templates

Chapter 21

Prefix an underscore for templates, which are snippets to be included, for example: _navbar.html The order of specifying template directories matters a lot. To better appreciate that, you need to understand how templates are rendered in Django.

How templates work Django renders templates while being agnostic of the actual template engine, as the following diagram shows:

Simplified depiction of template rendering in Django

Each template is rendered by trying each template backend specified by the TEMPLATES variable in settings.py in order. A Loader object corresponding to the backend will search for the template. Based on the backend's configuration, several kinds of loaders will be used. For instance, filesystem.Loader loads templates from the filesystem according to DIRS, and app_directories.Loader loads templates from within app directories.

[ 622 ]

Templates

Chapter 21

If a Loader is successful, the search ends and that particular backend template engine is chosen for rendering. This results in a Template object, which contains the parsed and compiled template. To render a Template, you will need to provide it with a Context object. Context behaves exactly like a dictionary, but is implemented as a stack of dictionaries. If a Template is a container for placeholders, then Context provides the values that fill these placeholders. While using Django Templates, you might be more familiar with RequestContext, which is a subclass of Context. A RequestContext adds more context to a template by running template context processors on the request. Jinja2 would not require context processors as it supports calling functions directly. Finally, the render method of a Template object receives the context and renders the output. This might be an HTML, XML, email, CSS, or any textual output. If you understand the template search order, then you can use it to your advantage to override the loaded templates. The following are some scenarios where this can comein handy: Override a third-party apps's template with your own project-defined template Use Jinja2 for performance-specific parts of your site and DTL for the rest The first one is a common use case due to the popularity of CSS frameworks such as Bootstrap. Madame O For the first time in weeks, Steve's office corner was bustling with frenetic activity. With more recruits, the now five-member team comprised of Brad, Evan, Jacob, Sue, and Steve. Like a superhero team, their abilities were deep and amazingly well-balanced. Brad and Evan were the coding gurus. While Evan was obsessed over details, Brad was the big-picture guy. Jacob's talent in finding corner cases made him perfect for testing. Sue was in charge of marketing and design. In fact, the entire design was supposed to be done by an avant-garde design agency. It took them a month to produce an abstract, vivid, color-splashed concept loved by the management. It took them

[ 623 ]

Templates

Chapter 21

another two weeks to produce an HTML-ready version from their Photoshop mockups. However, it was eventually discarded as it proved to be sluggish and awkward on mobile devices. Disappointed by the failure of what was now widely dubbed as the unicorn vomit design, Steve felt stuck. Hart had phoned him quite concerned about the lack of any visible progress to show management. In a grim tone, he reminded Steve, "We have already eaten up the project's buffer time. We cannot afford any last-minute surprises". It was then that Sue, who had been unusually quiet since she joined, mentioned that she had been working on a mockup using Twitter's Bootstrap. Sue was the growth hacker in the team — a keen coder and a creative marketer. She admitted having just rudimentary HTML skills. However, her mockup was surprisingly thorough and looked familiar to users of other contemporary social networks. Most importantly, it was responsive and worked perfectly on every device from tablets to mobiles. The management unanimously agreed on Sue's design, except for someone named Madame O. One Friday afternoon, she stormed into Sue's cabin and began questioning everything from the background color to the size of the mouse cursor. Sue tried to explain to her with surprising poise and calm. An hour later, when Steve decided to intervene, Madame O was questioning why the profile pictures had to be in a circle rather than a square. "But a site-wide change like that will never get over in time," he said. Madame O shifted her gaze to him and gave him a sly smile. Suddenly, Steve felt a wave of happiness and hope surged within him. It felt immensely relieving and stimulating. He heard himself happily agreeing to all she wanted. Later, Steve learnt that Madame Optimism was a minor mentalist who could influence prone minds. His team loved to bring up the latter fact on the slightest occasion.

[ 624 ]

Templates

Chapter 21

Using Bootstrap Hardly anyone designs an entire website from scratch these days. CSS frameworks such as Twitter's Bootstrap or Zurb's Foundation are easy starting points with grid systems, great typography, and preset styles. Most of them use responsive web design, making your site mobile friendly.

A website using modified Bootstrap Version 3.3 built using the Edge project skeleton

We will be using Bootstrap, but the steps will be similar for other CSS frameworks. There are three ways to include Bootstrap in your website: Find a project skeleton: If you have not yet started your project, then finding a project skeleton that already has Bootstrap is a great option. A project skeleton such as edge (created by yours truly) can be used as the initial structure while running startproject as follows:

[ 625 ]

Templates

Chapter 21 $ django-admin.py startproject -template=https://github.com/arocks/edge/archive/master.zip --extension=py,md,html myproj

Alternatively, you can use one of the cookiecutter templates with support for Bootstrap. Use a package: The easiest option if you have already started your project is to use a package, such as django-bootstrap4. Manually copy: None of the preceding options guarantees that their version of Bootstrap is the latest one. Bootstrap releases are so frequent that package authors have a hard time keeping their files up to date. So, if you would like to work with the latest version of Bootstrap, the best option is to download it from http://getbootstrap.com yourself. Be sure to read the release notes to check whether your templates need to be changed due to backward incompatibility. Copy the dist directory that contains the css, js, and fonts directories into your project root under the static directory. Ensure that this path is set for STATICFILES_DIRS in your settings.py: STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")]

Now you can include the Bootstrap assets in your templates, as follows: {% load staticfiles %}

But they all look the same! Bootstrap might be a great way to get started quickly. However, sometimes, developers get lazy and do not bother to change the default look. This leaves a poor impression on your users who might find your site's appearance a little too familiar and uninteresting. Bootstrap 4 comes with plenty of options to improve its visual appeal. You can

create a file called custom.scss where you can customize everything from theme colors to grid breakpoints. The documentation explains how you can set up the build system to compile these files down to the style sheets.

[ 626 ]

Templates

Chapter 21

Thanks to the huge community around Bootstrap, there are also several sites, such as bootswatch.com, which have themed style sheets, that are drop-in replacements for your bootstrap.min.css. Last but least and least, you can make your CSS classes more meaningful by replacing structural class names, such as row or col-lg-9, with semantic tags, such as main or article. You can do this with a few lines of SASS code to @extend the Bootstrap classes, as follows: @import "bootstrap"; body > main { @extend .row; article { @extend .col-lg-9; } }

This is possible due to a feature called mixins (sounds familiar?). With the SASS source files, Bootstrap can be completely customized to your needs.

Lightweight alternatives Older browsers used to be very inconsistent in how they handled CSS. They not only had vendor-specific prefixes such as -WebKit-transition but also had their own quirks. Newer browsers follow modern standards better. Now, we also have more powerful layout models such as flexbox, which reduce the complexity of code. All these have resulted in some very lightweight CSS frameworks. For instance, Pure.css is only 3.8 KB minified and gzipped, but packed with features. Similarly, mini.css designed with mobile devices and modern browsers in mind is under 7 KB gzipped. For comparison, Bootstrap is 25 KB, gzipped, with all modules included. While these lightweight frameworks might save some initial page load time, be sure to test them with all the different browsers your target users might use. Tools such as CanIUse.com can help by showing which features are supported across browsers and platforms. Bootstrap is quite good at maintaining backward compatibility with the widest range of clients.

[ 627 ]

Templates

Chapter 21

Template patterns Django's template language is quite simple. However, you can save a lot of time by following some elegant template design patterns. Let's take a look at some of them.

Pattern — template inheritance tree Problem: Templates need lots of common markup in several pages. Solution: Use template inheritance wherever possible and include snippets elsewhere.

Problem details Users expect pages of a website to follow a consistent structure. Certain interface elements, such as navigation menu, headers, and footers are seen in most web applications. However, it is cumbersome to repeat them in every template. Most templating languages have an include mechanism. The contents of another file, possibly a template, can be included at the position where it is invoked. This can get tedious in a large project. The sequence of the snippets to be included in every template would be mostly the same. The ordering is important and hard to check for mistakes. Ideally, we should be able to create a base structure. New pages ought to extend this base to specify only the changes or make extensions to the base content.

[ 628 ]

Templates

Chapter 21

Solution details Django templates have a powerful extension mechanism. Similar to classes in programming, a template can be extended through inheritance. However, for that to work, the base itself must be structured into blocks as follows:

Modular base templates can be extended by individual page templates giving flexibility and consistent layout

The base.html template is, by convention, the base structure for the entire site. This template will usually be well-formed HTML (that is, with a preamble and matching closing tags) that has several placeholders marked with the {% block tags %} tag. For example, a minimal base.html file looks as follows:

{% block heading %}Untitled{% endblock %} {% block content %} {% endblock %}

[ 629 ]

Templates

Chapter 21

There are two blocks here, heading and content, which can be overridden. You can extend the base to create specific pages that can override these blocks. For example, here is an About page: {% extends "base.html" %} {% block content %}

This is a simple About page

{% endblock %} {% block heading %}About{% endblock %}

We do not have to repeat the entire structure. We can also mention the blocks in any order. The rendered result will have the right blocks in the right places as defined in base.html. If the inheriting template does not override a block, then its parent's contents are used. In the preceding example, if the About template does not have a heading, then it will have the default heading of Untitled. You can insert the parent's contents explicitly using {{ block.super }}, which can be useful when you want to append or prepend to it. The inheriting template can be further inherited forming an inheritance chain. This pattern can be used as a common derived base for pages with a certain layout, for example, a single-column layout. A common base template can also be created for a section of the site, for example, Blog pages. Usually, all inheritance chains can be traced back to a common root, base.html; hence, the pattern's name: Template inheritance tree. Of course, this need not be strictly followed. The error pages 404.html and 500.html are usually not inherited and are stripped bare of most template tags to prevent further errors. Another way of achieving this might be to use context processors. You can create a context processor, which will add a context variable that can be used in all your templates globally. But this is not advisable for common markup such as sidebars as it violates the separation of concerns by moving presentation out of the template layer.

[ 630 ]

Templates

Chapter 21

Pattern — the active link Problem: The navigation bar is a common component in most pages. However, the active link needs to reflect the current page the user is on. Solution: Conditionally, change the active link markup by setting context variables or based on the request path.

Problem details The naïve way to implement the active link in a navigation bar is to manually set it in every page. However, this is neither DRY nor foolproof.

Solution details There are several solutions to determine the active link. Excluding JavaScript-based approaches, they can be mainly grouped into template-only and custom tag-based solutions.

A template-only solution By mentioning an active_link variable while including the snippet of the navigation template, this solution is both simple and easy to implement. In every template, you will need to include the following line (or inherit it): {% include "_navbar.html" with active_link='link2' %}

The _navbar.html file contains the navigation menu with a set of checks for the active_link variable: {# _navbar.html #}

Link 1 Link 2 Link 3

[ 631 ]

Templates

Chapter 21

Custom tags Django templates offer a versatile set of built-in tags. It is quite easy to create your own custom tag. Since custom tags live inside an app, create a templatetags directory inside an app. This directory must be a package, so it should have an (empty) __init__.py file. Next, write your custom template in an appropriately named Python file. For example, for this active link pattern, we can create a file called nav.py with the following contents: # app/templatetags/nav.py from django.core.urlresolvers import resolve from django.template import Library register = Library() @register.simple_tag def active_nav(request, url): url_name = resolve(request.path).url_name if url_name == url: return "active" return ""

This file defines a custom tag named active_nav. It retrieves the URL's path component from the request argument (say, /about/). Then, the resolve() function is used to look up the URL pattern's name (as defined in urls.py) from the path. Finally, it returns the string "active" only when the pattern's name matches the expected pattern name. The syntax for calling this custom tag in a template is {% active_nav request 'pattern_name' %}. Notice that the request needs to be passed in every page that this tag is used. Including a variable in several views can get cumbersome. Instead, we add a built-in context processor to TEMPLATE_CONTEXT_PROCESSORS in settings.py so that the request will be present in a request variable across the site, as follows: # settings.py [ 'django.core.context_processors.request', ]

[ 632 ]

Templates

Chapter 21

Now, all that remains is to use this custom tag in your template to set the active attribute: {# base.html #} {% load nav %}

• Active 1
• Active 2
• Active 3

Summary In this chapter, we looked at the features of Django templates. Since it is easy to change the templating language in Django, many people might consider replacing it. However, it is important to learn the design philosophy of the built-in template language before we seek alternatives. In the next chapter, we will look into one of the killer features of Django, that is, the admin interface, and how we can customize it.

[ 633 ]

22 Admin Interface In this chapter, we will discuss the following topics: Customizing admin Enhancing models for the admin admin best practices Feature flags Django's prominent feature is the admin interface, which makes it stand out from the competition. It is a built-in app that automatically generates a user interface to add and modify a site's content. For many, the admin is Django's killer app, automating the boring task of creating admin interfaces for the models in your project. The admin enables your team to add content and continue development at the same time. Once your models are ready and migrations have been applied, you just need to add a line or two to create its admin interface. Let's see how.

Using the admin interface In a newly generated project, the admin interface is enabled by default. After starting your development server, you will be able to see a login page when you navigate to http://127.0.0.1:8000/admin/.

Admin Interface

Chapter 22

If you have configured a superuser's credentials (or the credentials of any staff user), then you could log into the admin interface, as shown in the following screenshot:

Screenshot of Django administration in a new project

If you have used Django before, you'll notice that the appearance of the admin interface has improved, especially the SVG icons on high-DPI screens. It also uses responsive design, which works across all major mobile browsers. However, your models will not be visible here, unless you register the model with the admin site. This is defined in your app's admin.py. For instance, in sightings/admin.py, we register the Sighting model, as follows: from django.contrib import admin from . import models admin.site.register(models.Sighting)

The first argument to register specifies the model class to be added to the admin site. Here, the second argument to register, a ModelAdmin class, has been omitted, hence we will get a default admin interface for the post model. Let's see how to create and customize this ModelAdmin class.

[ 635 ]

Admin Interface

Chapter 22

The Beacon "Having coffee?" asked a voice from the corner of the pantry. Sue almost spilled her coffee. A tall man wearing a tight red and blue colored costume stood to smile with hands on his hips. The logo emblazoned on his chest said, in large type, Captain Obvious. "Oh, my God," said Sue as she wiped at the coffee stain with a napkin. "Sorry, I think I scared you," said Captain Obvious "What is the emergency?" "Isn't it obvious that she doesn't know?" said a calm female voice from above. Sue looked up to find a shadowy figure slowly descend from the open hall. Her face was partially obscured by her dark matted hair, which had a few grey streaks. "Hi Hexa!" said the Captain "But then, what was the message on SuperBook about?" Soon, they were all at Steve's office staring at his screen. "See, I told you there is no beacon on the front page," said Evan. "We are still developing that feature." "Wait," said Steve. "Let me log in through a nonstaff account." In a few seconds, the page refreshed and an animated red beacon appeared at the top, prominently positioned. "That's the beacon I was talking about!" exclaimed Captain Obvious. "Hang on a minute," said Steve. He pulled up the source files for the new features deployed earlier that day. A glance at the beacon feature branch code made it clear what went wrong: if switch_is_active(request, 'beacon') and not request.user.is_staff(): beacon.activate()

[ 636 ]

Admin Interface

Chapter 22

"Sorry everyone," said Steve. "There has been a logic error. Instead of turning this feature on only for staff, we inadvertently turned it on for everyone but staff. It is turned off now. Apologies for any confusion." "So, there was no emergency?" asked Captain with a disappointed look. Hexa put an arm on his shoulder and said "I am afraid not, Captain." Suddenly, there was a loud crash, and everyone ran to the hallway. A man had apparently landed in the office through one of the floor-to-ceiling glass walls. Shaking off shards of broken glass, he stood up. "Sorry, I came as fast as I could," he said. "Am I late to the party?" Hexa laughed. "No, Blitz. Been waiting for you to join," she said.

Enhancing models for the admin Here is an example that enhances the model's admin for better presentation and functionality. You can look at the difference between the two following screenshots to see how a few lines of code can make a lot of difference:

The default admin list view for the sightings model

[ 637 ]

Admin Interface

Chapter 22

After the admin customizations explained in this section are made, the same information will be presented in a much more accessible manner, as shown in the following screenshot:

The improved admin list view for the sightings model

The admin app is smart enough to figure out a lot of things from your model automatically. However, sometimes the inferred information can be improved. This usually involves adding an attribute or a method to the model itself (rather than to the ModelAdmin class). Here is the enhanced Sightings model: # models.py class Sighting(models.Model): superhero = models.ForeignKey( settings.AUTH_USER_MODEL, on_delete=models.CASCADE) power = models.CharField(max_length=100) location = models.ForeignKey(Location, on_delete=models.CASCADE) sighted_on = models.DateTimeField() def __str__(self): return "{}'s power {} sighted at: {} on {}".format( self.superhero, self.power, self.location.country,

[ 638 ]

Admin Interface

Chapter 22 self.sighted_on)

def get_absolute_url(self): from django.urls import reverse return reverse('sighting_details', kwargs={'pk': self.id}) class Meta: unique_together = ("superhero", "power") ordering = ["-sighted_on"] verbose_name = "Sighting & Encounter" verbose_name_plural = "Sightings & Encounters"

Let's take a look at how admin uses all these nonfield attributes: __str__(): Without this, the list of superhero entries would look

extremely boring. All entries would be shown alike, with the format of < Sighting: Sighting object>. Try to display the object's unique information in its str representation (or Unicode representation, in the case of Python 2.x code), such as its name or version. Anything that helps the admin to recognize the object unambiguously would help. get_absolute_url(): This method is handy if you like to switch between the admin site and the object's corresponding detail view on your (nonadmin) website. If this method is defined, then a button labeled View on site will appear in the top right-hand corner of the object's Edit page within the admin. ordering: Without this Meta option, your entries can appear in any order as returned from the database. As you can imagine, this is no fun for the admins if you have a large number of objects. The admins usually prefer to see fresh entries first, so sorting by date in the reverse chronological order (hence the minus sign) is common. verbose_name: If you omit this attribute, your model's name would be converted from CamelCase into camel case. In this case, it used frivolously to change "Sighting" to "Sighting & Encounter". But sometimes, the automatically generated verbose_name looks awkward, and you can specify how you would like the user-readable name to appear in the admin interface here. verbose_name_plural: Again, omitting this option can leave you with funny results. Since Django simply prepends an s to the word, the generated plural would be shown as "Sighting & Encounters" (on the admin front page, no less), so it is better to define it correctly here.

[ 639 ]

Admin Interface

Chapter 22

It is recommended that you define the previous Meta attributes and methods not just for the admin interface, but for better representation in the shell, log files, and so on. However, you can use many more features of the admin by creating a custom ModelAdmin class. In this case, we customize it as follows: # admin.py class SightingAdmin(admin.ModelAdmin): list_display = ('superhero', 'power', 'location', 'sighted_on') date_hierarchy = 'sighted_on' search_fields = ['superhero'] ordering = ['superhero']

admin.site.register(models.Sighting, SightingAdmin)

Let's take a look at these options more closely: list_display: This option shows the model instances in a tabular form. Instead of using the model's __str__ representation, it shows each field

mentioned as a separate sortable column. This is ideal if you like to sort by more than one attribute of your model. date_hierarchy: Specifying any date-time field of the model as a date hierarchy will present a date drill down (note the clickable years below the Search box). search_fields: This option shows a Search box above the list. Any search term entered would be searched against the mentioned fields. Hence, only text fields such as CharField or TextField can be mentioned here. ordering: This option takes precedence over your model's default ordering. It is useful if you prefer a different ordering in your admin screen, which is the preference we have adopted here. We have only mentioned a subset of the most commonly used admin options. Certain kinds of sites use the admin interface heavily. In such cases, it is highly recommended that you go through and understand the admin part of the Django documentation.

[ 640 ]

Admin Interface

Chapter 22

Not everyone should be an admin Since admin interfaces are so easy to create, people tend to misuse them. Some give users administration access indiscriminately by merely turning on their staff flag. Soon, users begin making feature requests, mistaking the admin interface for the actual application interface. Unfortunately, this is not what the admin interface is for. As the word staff suggests, it is an internal tool for the staff to enter content. It is production-ready, but not really intended for the end users of your website. It is best to use admin for simple data entry. For example, in a school-wide intranet project I once reviewed, every teacher was made an admin for a Django application. This was a poor decision since the admin interface confused the teachers. The workflow for scheduling a class involves checking the schedules of other teachers and students. Using the admin interface gives them a direct view of the database. There is very little control over how the data gets modified by the administrator. So, keep the set of people with admin access as small as possible. Make changes via admin sparingly, unless it is simple data entry, such as adding an article's content. Best Practice Don't give admin access to end users. Ensure that all your admins understand the data inconsistencies that can arise from making changes through the admin. If possible, record manually, or use apps, such as django-audit-log, that can keep a log of admin changes made for future reference. In the case of the university example, we created a separate interface for teachers, such as a course scheduler. These tools contain application code that can be used for purposes that are far beyond admin's data entry functionality, such as the detection of date conflicts. Essentially, rectifying most misuses of the admin interface involve creating more powerful tools for certain sets of users. However, don't take the easy (and wrong) path of granting them admin access.

[ 641 ]

Admin Interface

Chapter 22

Admin interface customizations The out-of-the-box admin interface is quite useful when getting started. Unfortunately, most people assume that it is quite hard to change the Django admin and leave it as it is. In fact, the admin is extremely customizable, and its appearance can be drastically changed with minimal effort.

Changing the heading Many users of the admin interface might be stumped by the heading—Django administration. It might be more helpful to change this to something customized, such as MySite Admin, or something cool, such as SuperBook Secret Area. It is quite easy to make this change. Simply add the following line to your site's urls.py: admin.site.site_header = "SuperBook Secret Area"

Changing the base and stylesheets Almost every admin page is extended from a common base template named admin/base_site.html. This means that with a little knowledge of HTML and CSS, you can make all sorts of customizations to change the look and feel of the admin interface. Create a directory called admin in any templates directory. Then, copy the base_site.html file from the Django source directory and alter it according to your needs. If you don't know where the templates are located, just run the following commands within the Django shell: >>> from os.path import join >>> from django.contrib import admin >>> print(join(admin.__path__[0], "templates", "admin")) /home/arun/env/sbenv/lib/python3.6/sitepackages/django/contrib/admin/templates/admin

The last line is the location of all your admin templates. You can override or extend any of these templates.

[ 642 ]

Admin Interface

Chapter 22

For an example of overriding the admin base template, you can change the font of the entire admin interface to Special Elite from Google Fonts, which is great for giving a mock-serious look. You will need to copy base_site.html from the admin templates to admin/base_site.html in one of your template's directories. Then, add the following lines to the end: {% block extrastyle %}

{% endblock %}

This adds an extra stylesheet for overriding the font-related styles and will be applied to every admin page.

Adding a rich-text editor for WYSIWYG editing Sometimes, you will need to include JavaScript code in the admin interface. A common requirement is to use an HTML editor, such as CKEditor, for your TextField. There are several ways to implement this in Django, for example, using a Media inner class on your ModelAdmin class. However, I find extending the admin change_form template to be the most convenient approach. For example, if you have an app called posts, then you will need to create a file called change_form.html within the templates/admin/posts/ directory. If you need to show CKEditor (it could be any JavaScript editor, but this one is the one I prefer) for the message field of a model in this app, then the contents of the file can be as follows: {% extends "admin/change_form.html" %} {% block footer %} {{ block.super }}

{% endblock %}

The part in bold is the automatically created ID for the form element we wish to enhance from a normal textbox to a rich-text editor. This change will not affect other textboxes or form fields in the admin site. These scripts and styles have been added to the footer block so that the form elements are created in the DOM before they are changed. Other approaches for achieving this might require the installation of apps and other configuration changes. For changing just one admin site field, this might be overkill. The approach here also gives you the flexibility to pick and choose the JavaScript editor of your choice.

Bootstrap-themed admin Unsurprisingly, a common request for admin customization is whether it can be integrated with Bootstrap. There are several packages that can do this, such as Django-admin-bootstrapped or Django suit. Rather than overriding all the admin templates yourself, these packages provide ready-to-use Bootstrap-themed templates. They are easy to install and deploy. Being based on Bootstrap, they are responsive and come with a variety of widgets and components.

Complete overhauls Attempts have been made to completely reimagine the admin interface. Grappelli is a very popular skin that extends the Django admin with new features, such as autocomplete lookups and collapsible inlines. With django-admin-tools, you get a customizable dashboard and menu bar.

[ 644 ]

Admin Interface

Chapter 22

Attempts have also been made to completely rewrite the admin, such as djangoadmin2 and nexus, which did not achieve any significant adoption. There is even an official proposal called AdminNext to revamp the entire admin app. Considering the size, complexity, and popularity of the existing admin, any such effort is expected to take a significant amount of time.

Protecting the admin The admin interface of your site provides access to almost every piece of data stored, so don't leave the metaphorical gate lightly guarded. In fact, one of the only telltale signs that someone is running Django is that when you navigate to http://example.com/admin/, you will be greeted by the blue login screen. In production, it is recommended that you change this location to something less obvious. It is as simple as changing the following line in your root urls.py: path('secretarea/', admin.site.urls),

A slightly more sophisticated approach is to use a dummy admin site at the default location or a honeypot (see the django-admin-honeypot package). However, the best option is to use HTTPS for your admin area (and everywhere else) since normal HTTP will send all the data in plain-text over the network. Check your web server documentation on how to set up HTTPS for admin requests (or, even better, if your entire site can be on HTTPS). On Nginx, it is quite easy to set this up. This involves specifying the SSL certificate locations. Finally, redirect all HTTP requests for admin pages to HTTPS, and you can sleep more peacefully. The following pattern is not strictly limited to the admin interface but it is nonetheless included in this chapter, as it is often controlled in the admin.

[ 645 ]

Admin Interface

Chapter 22

Pattern – feature flags Problem: The publishing of new features to users should be independent of the deployment of the corresponding code in production. Solution: Use feature flags to selectively enable or disable features after deployment.

Problem details Rolling out frequent bug fixes and new features to production is common today. Many of these changes are unnoticed by users. However, new features that have a significant impact in terms of usability or performance ought to be rolled out in a phased manner. In other words, deployment should be decoupled from a release. Simplistic release processes activate new features as soon as they are deployed. This can potentially have catastrophic results, ranging from user issues (swamping your support resources) to performance issues (causing downtime). Hence, in large sites, it is important to decouple deployment of new features in production and their activation. Even if they are activated, they are sometimes only seen by a select group of users. This select group can be staff or a limited set of customers who get an early preview.

Solution details Many sites control the activation of new features using feature flags. Typically, this is a switch controlled in each environment. A feature flipper is a switch in your code that determines whether a feature should be made available to certain customers. But we shall use the general term feature flags here. Several Django packages provide feature flags, such as gargoyle and django-waffle. These packages store feature flags of a site in the database. They can be activated or deactivated through the admin interface or through management commands. Hence, every environment (production, testing, development, and so on) can have its own set of activated features.

[ 646 ]

Admin Interface

Chapter 22

Feature flags were originally documented in Flickr (see http://code.flickr.net/2009/12/02/flipping-out/). They managed a code repository without any branches—that is, everything was checked into the mainline. They also deployed this code into production several times a day. If they found out that a new feature broke anything in production or increased load on the database, then they simply disabled it by turning that feature flag off. Feature flags can be used for various other situations (the following examples use Django Waffle): Trials: A feature flag can also be conditionally active for certain users. These can be your own staff or certain early adopters that you may be targeting, as follows: def my_view(request): if flag_is_active(request, 'flag_name'): # Behavior if flag is active.

Sites can run several such trials in parallel, so different sets of users might actually have different user experiences. Metrics and feedback are collected from these controlled tests before wider deployment. A/B testing: This is quite similar to trials, except that users are selected randomly within a controlled experiment. This method is quite common in web design and is used to identify which changes can increase the conversion rates. The following is how such a view can be written: def my_view(request): if sample_is_active(request, 'new_design'): # Behavior for test sample.

Performance testing: Sometimes, it is hard to measure the impact of a feature on server performance. In such cases, it is best to activate the flag only for a small percentage of users first. The percentage of activation can be gradually increased if the performance is within the expected limits.

[ 647 ]

Admin Interface

Chapter 22

Limit externalities: We can also use feature flags as a site-wide feature switch that reflects the availability of its services. For example, downtime in external services such as Amazon S3 can result in users facing error messages while they perform actions such as uploading photos. When the external service is down for extended periods, a feature flag can be deactivated and would disable the Upload button and/or show a more helpful message about the downtime. This simple feature saves the user's time and provides a better user experience: def my_view(request): if switch_is_active('s3_down'): # Disable uploads and show it is downtime

The main disadvantage of this approach is that the code gets littered with conditional checks. However, this can be controlled by periodic code cleanups that remove checks for fully accepted features and prune out permanently deactivated features. The activation of flags can be controlled from the admin site using the builtin user authentication and permissions systems. You can also control the sample percentage from the admin interface.

Summary In this chapter, we explored Django's built-in admin app. We found that it is not only quite useful out of the box, but that various customizations can also be made to improve its appearance and functionality. In the next chapter, we will take a look at how to use forms more effectively in Django by considering various patterns and common use cases.

[ 648 ]

23 Forms In this chapter, we will discuss the following topics: Form workflow Untrusted input Form processing with class-based views Working with CRUD views Let's set aside Django forms and talk about web forms in general. Forms are not just long, boring pages with several fields that you have to fill in. Forms are everywhere. We use them every day. Forms power everything from Google's search box to Facebook's Like button. Django abstracts most of the grunt work while working with forms such as validation or presentation. It also implements various security best practices. However, forms are also common sources of confusion because they could be in one of several states. Let's examine them more closely.

How forms work Forms can be tricky to understand because interacting with them takes more than one request-response cycle. In the simplest scenario, you need to present an empty form, which the user then fills in correctly and submits. Conversely, they might enter some invalid data, in which case the form needs to be resubmitted until the entire form is valid. From this scenario, we can see that a form can be one of several states, changing between them: Empty form (unfilled form): This form is called an unbound form in Django Filled form: This form is called a bound form in Django

Forms

Chapter 23

Submitted form with errors: This form is called a bound form but not a valid form Submitted form without errors: This form is called a bound and valid form The users will never see the form in the submitted form without errors state. They don't have to. Typically, submitting a valid form should take the users to a success page.

Forms in Django Django's form class instances contain the state of each field and, by summarizing them up a level, of the form itself. The form has two important state attributes, which are as follows: is_bound: If this returns false, then it is an unbound form, that is, a fresh

form with empty or default field values. If it returns true, then the form is bound, that is, at least one field has been set with a user input. is_valid(): If this returns true, then every field in the bound form has valid data. If false, then there is some invalid data in at least one field or the form is not bound. For example, imagine that you need a simple form that accepts a user's name and age. The forms class can be defined as follows (refer to the code in formschapter/forms.py): from django import forms class PersonDetailsForm(forms.Form): name = forms.CharField(max_length=100) age = forms.IntegerField()

This class can be initiated in a bound or unbound manner, as shown in the following code: >>> f = PersonDetailsForm() >>> print(f.as_p())

Name:

Age:

>>> f.is_bound

[ 650 ]

Forms

Chapter 23

False >>> g = PersonDetailsForm({"name": "Blitz", "age": "30"}) >>> print(g.as_p())

Name:

Age:

>>> g.is_bound True

Note how the HTML representation changes to include the value attributes with the bound data in them. The form can be bound only when you create the form object in the constructor. How does the user input end up in a dictionary-like object that contains values for each form field? To find this out, you need to understand how a user interacts with a form. In the following diagram, a user opens a person's details form, fills it incorrectly at first, submits it, and then resubmits it with the valid information:

Typical of submitting and processing a form

[ 651 ]

Forms

Chapter 23

As shown in the preceding diagram, when the user submits the form, the view callable gets all the form data inside request.POST (an instance of QueryDict). The form gets initialized with this dictionary-like object, referred to in this way as it behaves like a dictionary and has a bit of extra functionality. Forms can be defined so that they can send the form data in two different ways: GET or POST. Forms defined with METHOD="GET" send the form data encoded in the URL itself. For example, when you submit a Google search, your URL will have your form input, that is, the search string visibly embedded in the URL, such as ?q=Cat+Pictures. The GET method is used for idempotent forms, which do not make any lasting changes to the state of the world (or to be more pedantic, processing the form multiple times has the same effect as processing it once). For most cases, this means that it is used only to retrieve data. However, the vast majority of forms are defined with METHOD="POST". In this case, the form data is sent along with the body of the HTTP request, and it is not seen by the user. They are used for anything that involves a side effect, such as creating or updating data. Depending on the type of form you have defined, the view will receive the form data in request.GET or request.POST, when the user submits the form. As mentioned earlier, either of them will be like a dictionary, so you can pass it to your form class constructor to get a bound form object. The Breach Steve was curled up and snoring heavily in his large three-seater couch. For the last few weeks, he had been spending more than 12 hours at the office, and tonight was no exception. His phone lying on the carpet beeped. At first, he said something incoherent, still deep in sleep. Then, it beeped again and again, with increasing urgency. By the fifth beep, Steve awoke with a start. He frantically searched all over his couch, and finally located his phone on the floor. The screen showed a brightly colored bar chart. Every bar seemed to touch the top line except one. He pulled out his laptop and logged into the SuperBook server. The site was up and none of the logs indicated any unusual activity. However, the external services didn't look that good.

[ 652 ]

Forms

Chapter 23

The phone at the other end seemed to ring for eternity until a croaky voice answered, "Hello, Steve?". Half an hour later, Jacob was able to zero down the problem to an unresponsive superhero verification service. "Isn't that running on Sauron?" asked Steve. There was a brief hesitation. "I am afraid so," replied Jacob. Steve had a sinking feeling at the pit of his stomach. Sauron, a mainframe application, was their first line of defense against cyber attacks and other kinds of possible attack. It was three in the morning when he alerted the mission control team. Jacob kept chatting with him the whole time. He was running every available diagnostic tool. There was no sign of any security breach. Steve tried to calm him down. He reassured him that perhaps it was a temporary overload, and that he should get some rest. However, he knew that Jacob wouldn't stop until he found what was wrong. He also knew that it was not typical of Sauron to have a temporary overload. Feeling extremely exhausted, he slipped back to sleep. Next morning, as Steve hurried to his office building holding a bagel, he heard a deafening roar. He turned and looked up to see a massive spaceship looming over him. Instinctively, he ducked behind a hedge. On the other side of the hedge, he could hear several heavy metallic objects clanging onto the ground. Just then, his cell phone rang. It was Jacob. Something had moved closer to him. As Steve looked up, he saw a nearly 10-foot-tall robot, colored orange and black, pointing what looked like a weapon directly down at him. His phone was still ringing. He darted out into the open, barely missing the sputtering shower of bullets around him. He took the call. "Hey Steve, guess what, I found out what actually happened." "I am dying to know," Steve quipped. "Remember that we had used UserHoller's form widget to collect customer feedback? Apparently, their data was not that clean. I mean several serious exploits. Hey, there is a lot of background noise. Is that the TV?" Steve dived towards a large sign that said "Safe Assembly Point".

[ 653 ]

Forms

Chapter 23

"Just ignore it. Tell me what happened," he screamed. "Okay. So, when our admin opened the feedback page, his laptop must have gotten infected. The worm could reach the other systems he has access to, specifically, Sauron. I must say Steve, this is a very targeted attack. Someone who knows our security system quite well has designed this. I have a feeling something scary is coming our way." Across the lawn, a robot picked up an SUV and hurled it toward Steve. He raised his hands and shut his eyes. The spinning mass of metal froze a few feet above him. "Important call?" asked Hexa as she dropped the car. "Yeah, please get me out of here," Steve begged.

Why does data need cleaning? Eventually, you need to get the cleaned data from the form. Does this mean that the values that the user entered were not clean? Yes, for two reasons. First, anything that comes from the outside world should not be trusted initially. Malicious users can enter all sorts of exploits through a form that can undermine the security of your site. So, any form data must be sanitized before you use it. Best Practice Never trust the user input. Secondly, the field values in request.POST and request.GET are just strings. Even if your form field can be defined as an integer (say, age) or date (say, birthday), the browser would send them as strings to your view. Invariably, you would like to convert them to the appropriate Python types before use. The form class does this conversion automatically for you while cleaning.

[ 654 ]

Forms

Chapter 23

Let's see this in action: >>> fill = {"name": "Blitz", "age": "30"} >>> g = PersonDetailsForm(fill) >>> g.is_valid() True >>> g.cleaned_data {'age': 30, 'name': 'Blitz'} >>> type(g.cleaned_data["age"]) int

The age value was passed as a string (possibly from request.POST) to the form class. After validation, the cleaned data contains the age in the integer form. This is exactly what you would expect. Forms try to abstract away the fact that strings are passed around and give you clean Python objects that you can use. Always use the cleaned_data from your form rather than raw data from the user.

Displaying forms Django forms also help you create an HTML representation of your form. They support three different representations: as_p (as paragraph tags), as_ul (as unordered list items), and as_table (as, unsurprisingly, a table).

[ 655 ]

Forms

Chapter 23

The template code, generated HTML code, and browser rendering for each of these representations have been summarized in the following table: Template

Code

Output in Browser

Name:

}}

Age:

Name:

}}

Age:

Name: form.as_table Age:

Note that the HTML representation gives only the form fields. This makes it easier to include multiple Django forms in a single HTML form. However, this also means that the template designer has a fair bit of boilerplate to write for each form, as shown in the following code:

{% csrf_token %} {{ form.as_table }}

To make the HTML representation complete, you need to add the surrounding form tags, a csrf_token, the table or ul tags, and the Submit button.

[ 656 ]

Forms

Chapter 23

Time to be crisp It can get tiresome when writing so much boilerplate for each form in your templates. The django-crispy-forms package makes the form template code more crisp (that is, concise). It moves all the presentation and layout into the Django form itself. This way, you can write more Python code and less HTML. The following table shows that the crispy form template tag generates a more complete form, and the appearance is much more native to the Bootstrap style: Template Code

Output in Browser



crispy Name*

...

(HTML truncated for brevity)

So, how do you get crisper forms? You will need to install the django-crispyforms package and add it to your INSTALLED_APPS. If you use Bootstrap 4, then you will need to mention this in your settings: CRISPY_TEMPLATE_PACK = "bootstrap4"

The form initialization will need to mention a helper attribute of the FormHelper type. The following code in formschapter/forms.py is intended to be minimal and uses the default layout: from crispy_forms.helper import FormHelper from crispy_forms.layout import Submit class PersonDetailsForm(forms.Form): name = forms.CharField(max_length=100) age = forms.IntegerField() def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.helper = FormHelper(self) self.helper.layout.append(Submit('submit', 'Submit'))

For more details, read the django-crispy-forms package documentation.

[ 657 ]

Forms

Chapter 23

Understanding CSRF You must have noticed something called a cross-site request forgery (CSRF) token in the form templates. What does it do? It is a security mechanism against CSRF attacks for your forms. It works by injecting a server-generated random string called a CSRF token, unique to a user's session. Every time a form is submitted, it must have a hidden field that contains this token. This token ensures that the form was generated for the user by the original site, and proves that it is not a fake form created by an attacker with similar fields. CSRF tokens are not recommended for forms using the GET method because the GET actions should not change the server state. Moreover, forms submitted via GET would expose the CSRF token in the URLs. Since URLs have a higher risk of being logged or shoulder-sniffed, it is better to use CSRF in forms using the POST method.

Form processing with class-based views We can essentially process a form by subclassing the View class itself: class ClassBasedFormView(generic.View): template_name = 'form.html' def get(self, request): form = PersonDetailsForm() return render(request, self.template_name, {'form': form}) def post(self, request): form = PersonDetailsForm(request.POST) if form.is_valid(): # Success! We can use form.cleaned_data now return redirect('success') else: # Invalid form! Reshow the form with error highlighted return render(request, self.template_name, {'form': form})

Compare this code with the sequence diagram that we saw previously. The three scenarios have been separately handled.

[ 658 ]

Forms

Chapter 23

Every form is expected to follow the post/redirect/get (PRG) pattern. If the submitted form is found to be valid, then it must issue a redirect. This prevents duplicate form submissions. However, this is not a very DRY code. The form class name and template_name attributes have been repeated. Using a generic class-based view such as FormView can reduce the redundancy of form processing. The following code will give you the same functionality as the previous one, and in fewer lines of code: from django.urls import reverse_lazy class GenericFormView(generic.FormView): template_name = 'form.html' form_class = PersonDetailsForm success_url = reverse_lazy("success")

We need to use reverse_lazy in this case because the URL patterns are not loaded when the View file is imported.

Form patterns Let's take a look at some of the common patterns that are used when working with forms.

Pattern – dynamic form generation Problem: Adding form fields dynamically or changing form fields from what has been declared. Solution: Add or change fields during initialization of the form.

Problem details Forms are usually defined in a declarative style, with form fields listed as class fields. However, sometimes we do not know the number or type of these fields in advance. This calls for the form to be dynamically generated. This pattern is sometimes called dynamic form or runtime form generation.

[ 659 ]

Forms

Chapter 23

Imagine a passenger check-in system for a flight from an airport. The system allows for the upgrade of economy-class tickets to first class. If there are any first-class seats left, then it should show an additional option to the user, asking whether they would like to upgrade to first class. However, this optional field cannot be declared since it will not be shown to all users. Such dynamic forms can be handled by this pattern.

Solution details Every form instance has an attribute called fields, which is a dictionary that holds all the form fields. This can be modified at runtime. Adding or changing the fields can be done during form initialization itself. For example, if we need to add a checkbox to a user-details form only if a keyword argument named "upgrade" is true upon form initialization, then we can implement it as follows: class PersonDetailsForm(forms.Form): name = forms.CharField(max_length=100) age = forms.IntegerField() def __init__(self, *args, **kwargs): upgrade = kwargs.pop("upgrade", False) super().__init__(*args, **kwargs) # Show first class option? if upgrade: self.fields["first_class"] = forms.BooleanField( label="Fly First Class?")

Now, we just need to pass the PersonDetailsForm(upgrade=True) keyword argument to make an additional Boolean input field (a checkbox) appear. A newly introduced keyword argument has to be removed or popped before we call super to avoid the unexpected keyword error.

[ 660 ]

Forms

Chapter 23

If we use a FormView class for this example, then we need to pass the keyword argument by overriding the get_form_kwargs method of the View class, as shown in the following code: class PersonDetailsEdit(generic.FormView): ... def get_form_kwargs(self): kwargs = super().get_form_kwargs() kwargs["upgrade"] = True return kwargs

This pattern can be used to change any attribute of a field at runtime, such as its widget or help text. It works for model forms as well. In many cases, a seeming need for dynamic forms can be solved using Django formsets. They are used when a form needs to be repeated in a page. A typical use case for formsets is when designing a data-grid-like view to add elements row by row. This way, you do not need to create a dynamic form with an arbitrary number of rows; you just need to create a form for the row and create multiple rows using a formset_factory function.

Pattern – user-based forms Problem: Forms need to be customized based on the logged-in user. Solution: Pass the logged-in user's characteristics as a keyword argument to the form's initializer.

Problem details A form can be presented in different ways based on the user. Certain users might not need to fill in all the fields, while certain others might need to add additional information. In some cases, you might need to run some checks on the user's eligibility, such as verifying whether they are members of a group, to determine how the form should be constructed.

[ 661 ]

Forms

Chapter 23

Solution details As you must have noticed, you can solve this using the solution given in the dynamic form generation pattern. You just need to pass request.user or any of their characteristics as a keyword argument to the form. I would recommend the latter to minimize the coupling between the view and the form. As in the previous example, we need to show an additional checkbox to the user. However, this will be shown only if the user is a member of the "VIP" group. Let's take a look at how the GenericFormView derived view passes this information to the form: class GenericFormView(generic.FormView): template_name = 'cbv-form.html' form_class = PersonDetailsForm success_url = reverse_lazy("home") def get_form_kwargs(self): kwargs = super().get_form_kwargs() # Check if the logged-in user is a member of "VIP" group kwargs["vip"] = self.request.user.groups.filter( name="VIP").exists() return kwargs

Here, we are redefining the get_form_kwargs method that FormView calls before instantiating a form to return the keyword arguments. This is the ideal point to check whether the user belongs to the VIP group and pass the appropriate keyword argument. As before, the form can check for the presence of the vip keyword argument (like we did for upgrade) and present a check box for upgrading to first class.

Pattern – multiple form actions per view Problem: Handling multiple form actions in a single view or page. Solution: Forms can use separate views to handle form submissions, or a single view can identify the form based on the Submit button's name.

[ 662 ]

Forms

Chapter 23

Problem details Django makes it relatively straightforward to combine multiple forms with the same action, like a single Submit button. However, most web pages need to show several actions on the same page. For example, you might want the user to subscribe or unsubscribe from a newsletter using two distinct forms that are shown on the same page. However, Django's FormView is designed to handle only one form per view scenario. Many other generic class-based views also share this assumption.

Solution details There are two ways to handle multiple forms: using separate views and using a single view. Let's take a look at the first approach.

Separate views for separate actions This is a fairly straightforward approach, with each form specifying a different view as its action. For example, take the subscribe and unsubscribe forms. There can be two separate view classes to handle just the POST method from their respective forms.

Same view for separate actions Perhaps you find splitting the views to handle forms to be unnecessary, or you find handling logically related forms in a common view to be more elegant. Either way, we can work around the limitations of generic class-based views to handle more than one form. While using the same view class for multiple forms, the challenge is to identify which form issued the POST action. Here, we take advantage of the fact that the name and value of the Submit button is also submitted. If the Submit button is named uniquely across forms, then the form can be identified while processing. Here, we define a SubscribeForm using crispy forms so that we can name the Submit button as well: class SubscribeForm(forms.Form): email = forms.EmailField() def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.helper = FormHelper(self)

[ 663 ]

Forms

Chapter 23

self.helper.layout.append(Submit('subscribe_butn', 'Subscribe'))

The UnSubscribeForm class is defined in exactly the same way (and hence is omitted), except that its Submit button is named unsubscribe_butn. Since FormView is designed for a single form, we will use a simpler class-based view, say TemplateView, as the base for our view. Let's take a look at the view definition and the get method: from .forms import SubscribeForm, UnSubscribeForm class NewsletterView(generic.TemplateView): subcribe_form_class = SubscribeForm unsubcribe_form_class = UnSubscribeForm template_name = "newsletter.html" def get(self, request, *args, **kwargs): kwargs.setdefault("subscribe_form", self.subcribe_form_class()) kwargs.setdefault("unsubscribe_form", self.unsubcribe_form_class()) return super().get(request, *args, **kwargs)

The two forms are inserted as keyword arguments, and thereby enter the template context. We create unbound instances of either form only if they don't already exist, with the help of the setdefault dictionary method. We will soon see why. Next, we will take a look at the POST method, which handles submissions from either form: def post(self, request, *args, **kwargs): form_args = { 'data': self.request.POST, 'files': self.request.FILES, } if "subscribe_butn" in request.POST: form = self.subcribe_form_class(**form_args) if not form.is_valid(): return self.get(request, subscribe_form=form) return redirect("success_form1") elif "unsubscribe_butn" in request.POST: form = self.unsubcribe_form_class(**form_args) if not form.is_valid(): return self.get(request, unsubscribe_form=form)

[ 664 ]

Forms

Chapter 23 return redirect("success_form2") return super().get(request)

First, the form keyword arguments, such as data and files, are populated in a form_args dictionary. Next, the presence of the first form's Subscribe button is checked in request.POST. If the button's name is found, then the first form is instantiated. If the form fails validation, then the response created by the GET method with the first form's instance is returned. In the same way, we look for the second form's Unsubscribe button to check whether the second form was submitted. Instances of the same form in the same view can be implemented in the same way with form prefixes. You can instantiate a form with a prefix argument such as SubscribeForm(prefix="offers"). Such an instance will prefix all its form fields with the given argument, effectively working like a form namespace. In general, you can use prefixes to embed multiple forms in the same page.

Pattern – CRUD views Problem: Writing boilerplate for CRUD interfaces for a model becomes repetitive. Solution: Use generic class-based editing views.

Problem details In conventional web applications, most of the time is spent writing CRUD interfaces to a database. For instance, Twitter essentially involves creating and reading each other's tweets. Here, a tweet would be the database object that is being manipulated and stored. Writing such interfaces from scratch can get tedious. This pattern can be easily managed if CRUD interfaces can be automatically created from the model class itself.

[ 665 ]

Forms

Chapter 23

Solution details Django simplifies the process of creating CRUD views with a set of four generic classbased views. They can be mapped to their corresponding operations as follows: CreateView: This view displays a blank form to create a new model

instance DetailView: This view shows an object's details by reading from the database UpdateView: This view allows you to update an object's details through a prepopulated form DeleteView: This view displays a confirmation page and, on approval, deletes the object from the database Let's take a look at a simple example. We have a model that contains important dates about events of interest to everyone using our site. We need to build simple CRUD interfaces so that anyone can view and modify these dates. Let's take a look at the ImportantDate model defined in formschapter/models.py as follows: class ImportantDate(models.Model): date = models.DateField() desc = models.CharField(max_length=100) def get_absolute_url(self): return reverse('impdate_detail', args=[str(self.pk)])

The get_absolute_url() method is used by the CreateView and UpdateView classes to redirect after a successful object creation or update. It has been routed to the object's DetailView. The CRUD views themselves are simple enough to be self-explanatory, as shown in the following code within formschapter/views.py: class ImpDateDetail(generic.DetailView): model = models.ImportantDate

class ImpDateCreate(generic.CreateView): model = models.ImportantDate form_class = ImportantDateForm

class ImpDateUpdate(generic.UpdateView): model = models.ImportantDate form_class = ImportantDateForm

[ 666 ]

Forms

Chapter 23

class ImpDateDelete(generic.DeleteView): model = models.ImportantDate success_url = reverse_lazy("formschapter:impdate_list")

In these generic views, the model class is the only mandatory member to be mentioned. However, in the case of DeleteView, the success_url function needs to be mentioned as well. This is because after deletion, get_absolute_url can no longer be used to find out where to redirect users. Defining the form_class attribute is not mandatory. If it is omitted, a ModelForm method corresponding to the specified model will be created. However, we would like to create our own model form to take advantage of crispy forms, as shown in the following code in formschapter/forms.py: from from from from

django import forms . import models crispy_forms.helper import FormHelper crispy_forms.layout import Submit

class ImportantDateForm(forms.ModelForm): class Meta: model = models.ImportantDate fields = ["date", "desc"] def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.helper = FormHelper(self) self.helper.layout.append(Submit('save', 'Save'))

Thanks to crispy forms, we need very little HTML markup in our templates to build these CRUD forms. Explicitly mentioning the fields of a ModelForm method is a best practice. Setting fields to '__all__' may be convenient, but can inadvertently expose sensitive data, especially after adding new fields to the model.

[ 667 ]

Forms

Chapter 23

The template paths, by default, are based on the view class and the model names. For brevity, we omitted the template source here. Please refer to the templates directory in the formschapter app in the SuperBook project. We use the same form for CreateView and UpdateView. Finally, we take a look at formschapter/urls.py, where everything is wired up together: path('impdates//', views.ImpDateDetail.as_view(), name="impdate_detail"), path('impdates/create/', views.ImpDateCreate.as_view(), name="impdate_create"), path('impdates//edit/', views.ImpDateUpdate.as_view(), name="impdate_update"), path('impdates//delete/', views.ImpDateDelete.as_view(), name="impdate_delete"), path('impdates/', views.ImpDateList.as_view(), name="impdate_list"),

Django generic views are a great way to get started with creating CRUD views for your models. With a few lines of code, you get well-tested model forms and views created for you, rather than doing the boring task yourself.

Summary In this chapter, we looked at how web forms work and how they are abstracted using form classes in Django. We also looked at the various techniques and patterns that are used to save time while working with forms. In the next chapter, we will take a look at a systematic approach to work with a legacy Django codebase, and how we can enhance it to meet evolving client needs.

[ 668 ]

24 Security In this chapter, we will discuss the following topics: Various web attacks and countermeasures Where Django can and cannot help Security checks for Django applications Several prominent industry reports suggest that websites and web applications remain one of the primary targets of cyber attacks. Yet, about 86 percent of all websites, tested by a leading security firm in 2013, had at least one serious vulnerability. Releasing your application to the wild is fraught with several dangers ranging from the leaking of confidential information to denial-of-service attacks. Mainstream media headlines security flaws focusing on exploits, such as Heartbleed, Cloudbleed, Superfish, and POODLE, that have an adverse impact on critical website applications, such as email and banking. Indeed, one often wonders if WWW now means the World Wide Web or the Wild Wild West. One of the biggest selling points of Django is its strong focus on security. In this chapter, we will cover the top techniques that attackers use. As we will soon see in this chapter, Django can protect you from most of them out of the box. I believe that in order to protect your site from attackers, you will need to think like one. So, let's familiarize ourselves with the common attacks.

Cross-site scripting Cross-site scripting (XSS), considered the most prevalent web application security flaw today, enables an attacker to execute their malicious scripts (usually JavaScript) on web pages viewed by users. Typically, the server is tricked into serving their malicious content along with the trusted content.

Security

Chapter 24

How does a malicious piece of code reach the server? The common means of entering external data into a website are as follows: Form fields URLs Redirects External scripts such as Ads or Analytics None of these can be entirely avoided. The real problem is when outside data gets used without being validated or sanitized (as shown in the following screenshot); never trust outside data:

For example, let's take a look at a piece of vulnerable code and how an XSS attack can be performed on it. It is strongly advised that you do not to use this code in any form: class XSSDemoView(View): def get(self, request): # WARNING: This code is insecure and prone to XSS attacks # *** Do not use it!!! *** if 'q' in request.GET: return HttpResponse("Searched for: {}".format( request.GET['q'])) else: return HttpResponse("""

[ 670 ]

Security

Chapter 24

Go """)

The preceding code is a View class that shows a search form when accessed without any GET parameters. If the search form is submitted, it shows the Search string exactly as entered by the user in the form. Now, open this view in a dated browser (say, IE 8) and enter the following search term in the form and submit it:

Unsurprisingly, the browser will show an alert box with the ominous message pwned. This attack fails in current browsers such as the latest Chrome, which will present the following error message in the console: Refused to execute a JavaScript script. The source code of script found within request. In case you are wondering what harm a simple alert message could cause, remember that any JavaScript code can be executed in the same manner. In the worst case, the user's cookies can be sent to a site controlled by the attacker by entering the following search term:

Once your cookies are sent, the attacker might be able to conduct a more serious attack.

Why are your cookies valuable? It might be worth understanding why cookies are the target of several attacks. Simply put, access to cookies allows attackers to impersonate you and even take control of your web account. To understand this in detail, you need to understand the concept of sessions. HTTP is stateless. Be it an anonymous or an authenticated user, Django keeps track of their activities for a certain duration of time by managing sessions.

[ 671 ]

Security

Chapter 24

A session consists of a session ID at the client end, that is, the browser and a dictionary-like object stored at the server end. The session ID is a random 32-character string that is stored as a cookie in the browser. Each time a user makes a request to a website, all their cookies, including this session ID, are sent along with the request. At the server end, Django maintains a session store that maps this session ID to the session data. By default, Django stores the session data in the django_session database table. Once a user successfully logs in, the session will note that the authentication was successful and will keep track of the user. Therefore, the cookie becomes a temporary user authentication for subsequent transactions. Anyone who acquires this cookie can use this web application as that user, which is called session hijacking.

How Django helps You might have observed that my example was an extremely unusual way of implementing a view in Django for two reasons: it did not use templates for rendering, and form classes were not used. Both of them have XSS-prevention measures. By default, Django templates auto-escape HTML special characters. So, if you had displayed the search string in a template, all the tags would have been HTML encoded. This makes it impossible to inject scripts unless you explicitly turn them off by marking the content as safe. Using form classes in Django to validate and sanitize the input is also a very effective countermeasure. For example, if your application requires a numeric employee ID, then use an IntegerField class rather than the more permissive CharField class. In our example, we can use a RegexValidator class in our search-term field to restrict the user to alphanumeric characters and allow punctuation symbols recognized by your search module. Restrict the acceptable range of the user input as strictly as possible.

[ 672 ]

Security

Chapter 24

Where Django might not help Django can prevent 80 percent of XSS attacks through auto-escaping in templates. For the remaining scenarios, you must take care to do the following tasks: Quote all HTML attributes, for example, replace with Escape dynamic data in CSS or JavaScript using custom methods Validate all URLs, especially against unsafe protocols such as JavaScript Avoid client-side XSS (also, known as DOM-based XSS) As a general rule against XSS, I suggest filter on input and escape on output. Make sure that you strictly validate and sanitize (filter) any data that comes in and transform (escape) it immediately before sending it to the user—specifically, if you need to support the user input with HTML formatting such as comments, consider using Markdown instead. Filter on input and escape on output.

Cross-site request forgery Cross-site request forgery (CSRF) is an attack that tricks a user into making unwanted actions on a website, where they are already authenticated, while they are visiting another site. Say, in a forum, an attacker can place an IMG or IFRAME tag within the page that makes a carefully crafted request to the authenticated site. For instance, the following fake 0x0 image can be embedded in a comment:

If you have already signed into SuperBook from another tab, and if the site doesn't have CSRF countermeasures, then a very embarrassing message will be posted. In other words, CSRF allows the attacker to perform actions by assuming your identity.

[ 673 ]

Security

Chapter 24

How Django helps The basic protection against CSRF is to use an HTTP POST (or PUT and DELETE, if supported) for any action that has side effects. Any GET (or HEAD) request must be used for information retrieval, for example, read-only. Django offers countermeasures against POST, PUT, or DELETE methods by embedding a token. You must already be familiar with the {% csrf_token %} mentioned inside each Django form template. This is rendered into a random value that must be present while submitting the form. The way this works is that the attacker will not be able to guess the token while crafting the request to your authenticated site. Since the token is mandatory and must match the value presented while displaying the form, the form submission fails and the attack is thwarted.

Where Django might not help Some people turn off CSRF checks in a view with the @csrf_exempt decorator, especially for AJAX form posts. This is not recommended unless you have carefully considered the security risks involved.

SQL injection SQL injection is the second most common vulnerability of web applications, after XSS. The attack involves entering malicious SQL code into a query that gets executed on the database. It could result in data theft, by dumping database content, or the destruction of data, say, by using the DROP TABLE command. If you are familiar with SQL, then you can understand the following piece of code; it looks up an email address based on the given username: name = request.GET['user'] sql = "SELECT email FROM users WHERE username = '{}';".format(name)

[ 674 ]

Security

Chapter 24

At first glance, it might appear that only the email address corresponds to the username mentioned as the GET parameter will be returned. However, imagine if an attacker entered ' OR '1'='1' in the form field, then the SQL code would be as follows: SELECT email FROM users WHERE username = '' OR '1'='1';

Since this WHERE clause will always be true, the emails of all the users of your application will be returned. This can be a serious leak of confidential information. Again, if the attacker wishes, they could execute more dangerous queries like the following: SELECT email FROM users WHERE username = ''; DELETE FROM users WHERE '1'='1';

Now, all the user entries will be wiped off your database!

How Django helps The countermeasure against an SQL injection is fairly simple. Use the Django ORM rather than crafting SQL statements by hand. The preceding example should be implemented as follows: User.objects.get(username=name).email

Here, Django's database drivers will automatically escape the parameters. This will ensure that they are treated as purely data and, therefore, they are harmless. However, as we will soon see, even the ORM has a few escape latches.

Where Django might not help There could be instances where people would need to resort to raw SQL, say, due to limitations of the Django ORM. For example, the where clause of the extra() method of a QuerySet allows raw SQL. This SQL code will not be escaped against SQL injections.

[ 675 ]

Security

Chapter 24

If you are using the low-level ORM API, such as the execute() method, then you might want to pass bind parameters instead of interpolating the SQL string yourself. Even then, it is strongly recommended that you check whether each identifier has been properly escaped. Finally, if you are using a third-party database API such as MongoDB, then you will need to manually check for SQL injections. Ideally, you would want to use only thoroughly sanitized data with such interfaces.

Clickjacking Clickjacking is a means of misleading a user to click on a hidden link or button in the browser when they were intending to click on something else. This is typically implemented using an invisible IFRAME that contains the target website over a dummy web page (shown here) that the user is likely to click on:

Since the action button in the invisible frame would be aligned exactly above the button in the dummy page, the user's click will perform an action on the target website instead.

[ 676 ]

Security

Chapter 24

How Django helps Django protects your site from clickjacking using middleware that can be fine-tuned using several decorators. By default, this django.middleware.clickjacking.XFrameOptionsMiddleware middlewar e will be included in your MIDDLEWARE_CLASSES within your settings file. It works by setting the X-Frame-Options header to SAMEORIGIN for every outgoing HttpResponse. Most modern browsers recognize the header, which means that this page should not be inside a frame in other domains. The protection can be enabled and disabled for certain views using decorators, such as @xframe_options_deny and @xframe_options_exempt.

Shell injection As the name suggests, shell injection or command injection allows an attacker to inject malicious code into a system shell such as bash. Even web applications use command-line programs for convenience and their functionality. Such processes are typically run within a shell. For example, if you want to show all the details of a file whose name is given by the user, a naïve implementation would be as follows: os.system("ls -l {}".format(filename))

An attacker can enter the filename as manage.py; rm -rf * and delete all the files in your directory. In general, it is not advisable to use os.system. The subprocess module is a safer alternative (or even better, you can use os.stat() to get the file's attributes). Since a shell will interpret the command-line arguments and environment variables, setting malicious values in them can allow the attacker to execute arbitrary system commands.

How Django helps Django primarily depends on WSGI for deployment. Since WSGI, unlike CGI, does not set on environment variables based on the request, the framework itself is not vulnerable to shell injections in its default configuration.

[ 677 ]

Security

Chapter 24

However, if the Django application needs to run other executables, then care must be taken to run it in a restricted manner, that is, with least permissions. Any parameter originating externally must be sanitized before passing to such executables. Additionally, use call() from the subprocess module to run command-line programs with its default shell=False parameter to handle arguments securely if shell interpolation is not necessary.

And the web attacks are unending There are hundreds of attack techniques that we have not covered here, and the list keeps growing every day as new attacks are found. It is important to keep ourselves aware of them. Django's official blog (https://www.djangoproject.com/weblog/) is a great place to find out about the latest exploits that have been discovered. Django maintainers proactively try to resolve them by releasing security releases. It is highly recommended that you install them as quickly as possible since they usually need very little or no changes to your source code. The security of your application is only as strong as its weakest link. Even if your Django code might be completely secure, there are so many layers and components in your stack, not to mention human elements, who can also be tricked with various social engineering techniques, such as phishing. Vulnerabilities in one area, such as the OS, database, or web server, can be exploited to gain access to other parts of your system. Hence, it is best to have a holistic view of your stack rather than view each part separately. The safe room As soon as Steve stepped outside the boardroom, he took out his phone and thumbed a crisp one-liner e-mail to his team: "It's a go!" In the last 60 minutes, he had been grilled by the directors on every possible detail of the launch. Madam O, to Steve's annoyance, maintained her stoic silence the entire time. He entered his cabin and opened his slide printouts once more. The number of trivial bugs dropped sharply after the checklists were introduced. Essential features that were impossible to include in the release were worked out through early collaboration with helpful users, such as Hexa and Aksel.

[ 678 ]

Security

Chapter 24

The number of signups for the beta site had crossed 9,000, thanks to Sue's brilliant marketing campaign. Never in his career had Steve seen so much interest for a launch. It was then that he noticed something odd about the newspaper on his desk. Fifteen minutes later, he rushed down the aisle in level 21. At the very end, there was a door marked 2109. When he opened it, he saw Evan working on what looked like a white plastic toy laptop. "Why did you circle the crossword clues? You could have just called me," asked Steve. "I want to show you something," he replied with a grin. He grabbed his laptop and walked out. He stopped between room 2110 and the fire exit. He fell on his knees and with his right hand, he groped the faded wallpaper. "There has to be a latch here somewhere," he muttered. Then, his hand stopped and turned a handle barely protruding from the wall. A part of the wall swiveled and came to a halt. It revealed an entrance to a room lit with a red light. A sign inside dangling from the roof said "Safe room 21B." As they entered, numerous screens and lights flicked on by themselves. A large screen on the wall said "authentication required. Insert key." Evan admired this briefly and began wiring up his laptop. "Evan, what are we doing here?" asked Steve in a hushed voice. Evan stopped, "Oh, right. I guess we have some time before the tests finish." He took a deep breath. "Remember when Madam O wanted me to look into the Sentinel codebase? I did. I realized that we were given censored source code. I mean I can understand removing some passwords here and there, but thousands of lines of code? I kept thinking-there had to be something going on." "So, with my access to the archiver, I pulled some of the older backups. The odds of not erasing a magnetic medium are surprisingly high. Anyways, I could recover most of the erased code. You won't believe what I saw."

[ 679 ]

Security

Chapter 24

Sentinel was not an ordinary social network project. It was a surveillance program. Perhaps the largest known to mankind. Post-Cold War, a group of nations joined to form a network to share intelligence information. A network of humans and sentinels. Sentinels are semi-autonomous computers with unbelievable computing power. Some believe they are quantum computers. Sentinels were inserted at thousands of strategic locations around the world-mostly ocean beds where major fiber optic cables are passed. Running on geothermal energy, they were self–powered and practically indestructible. They had access to nearly every internet communication in most countries. At some point in the nineties, perhaps fearing public scrutiny, the Sentinel program was shut down. This is where it gets really interesting. The code history suggests that the development on Sentinels was continued by someone named Cerebos. The code has been drastically enhanced from its surveillance abilities to form a sort of massively parallel supercomputer. A number-crunching beast for whom no encryption algorithm poses a significant challenge. Remember the breach? I found it hard to believe that there was not a single offensive move before the superheroes arrived. So, I did some research. SHIM's cybersecurity is designed as five concentric rings. We, the employees, are in the outermost, least privileged, ring protected by Sauron. Inner rings are designed with increasingly stronger cryptographic algorithms. This room is in level 4. My guess is that long before we knew about the breach, all systems of Sauron were already compromised. Systems were down and it was practically a cakewalk for those robots to enter the campus. I just looked at the logs. The attack was extremely targeted–everything from IP addresses to logins were known beforehand.

[ 680 ]

Security

Chapter 24

"Insider?" asked Steve in horror. "Yes. However, Sentinels needed help only for Level 5. Once they acquired the public keys for Level 4, they began attacking Level 4 systems. It sounds insane but that was their strategy." "Why is it insane?" "Well, most of the world's online security is based on public-key cryptography or asymmetric cryptography. It is based on two keys: one public and the other private. Although mathematically related, it is computationally impractical to find one key if you have the other." "Are you saying that the Sentinel network can?" "In fact, they can for smaller keys. Based on the tests I am running right now, their powers have grown significantly. At this rate, they should be ready for another attack in less than 24 hours." "Damn, that's when SuperBook goes live!"

A handy security checklist Security is not an afterthought but is instead integral to the way you write applications. However, being human, it is handy to have a checklist to remind you of the common omissions. The following points are a bare minimum of security checks that you should perform before making your Django application public: Don't trust data from a browser, API, or any outside sources: This is a fundamental rule. Make sure that you validate and sanitize any outside data. Don't keep SECRET_KEY in version control: As a best practice, pick SECRET_KEY from the environment. Check out the djangoenviron package.

[ 681 ]

Security

Chapter 24

Don't store passwords in plain text: Store your application password hashes instead. Add a random salt as well. Don't log any sensitive data: Filter out the confidential data, such as credit card details or API keys, before recording them in your log files. Any secure transaction or login should use SSL: Be aware that eavesdroppers in the same network as you could listen to your web traffic if it is not in HTTPS. Ideally, you ought to use HTTPS for the entire site. Avoid using redirects to user-supplied URLs: If you have redirects such as http://example.com/r?url=http://evil.com, then always check against whitelisted domains. Check authorization even for authenticated users: Before performing any change with side effects, check whether the logged-in user is allowed to perform it. Use the strictest possible regular expressions: Be it your URLconf or form validators, you must avoid lazy and generic regular expressions. Don't keep your Python code in web root: This can lead to an accidental leak of source code if it gets served as plain text. Use Django templates instead of building strings by hand: Templates have protection against XSS attacks. Use Django ORM rather than SQL commands: The ORM offers protection against SQL injection. Use Django forms with POST input for any action with side effects: It might seem like overkill to use forms for a simple vote button, but do it. CSRF should be enabled and used: Be very careful if you are exempting certain views using the @csrf_exempt decorator. Ensure that Django and all packages are the latest versions: Plan for updates. They might need some changes to be made to your source code. However, they bring shiny new features and security fixes too. Limit the size and type of user-uploaded files: Allowing large file uploads can cause denial-of-service attacks. Deny uploading of executables or scripts. Have a backup and recovery plan: Thanks to Murphy, you can plan for an inevitable attack, catastrophe, or any other kind of downtime. Make sure that you take frequent backups to minimize data loss.

[ 682 ]

Security

Chapter 24

Some of these can be checked automatically using Erik's Pony Checkup at http://ponycheckup.com/. However, I would recommend that you print or copy this checklist and stick it on your desk. Remember that this list is by no means exhaustive and not a substitute for a proper security audit by a professional.

Summary In this chapter, we looked at the common types of attacks affecting websites and web applications. In many cases, the explanation of the techniques has been simplified for clarity at the cost of detail. However, once we understand the severity of the attack, we can appreciate the countermeasures that Django provides. In our final chapter, we will take a look at predeployment activities in more detail. We will also take a look at the various deployment strategies, such as cloud-based hosting for deploying a Django application.

[ 683 ]

25 Working Asynchronously In this chapter, we will cover the following topics: Need for asynchronous Asynchronous patterns Working with Celery Understanding asyncio Entering channels In simpler times, a web application used to be a large monolithic Django process that can handle a request and block until the response is generated. In today's microservices world, applications are made up of a complex and ofteninterlocking chain of processes providing specialized services. Django is possibly one of the links in an application flow. As Eliyahu Goldratt would say, "the chain is only as strong as its weakest link". In other words, the synchronous nature of Django can potentially make it a performance bottleneck. Hence, there are various asynchronous solutions built around Django that can help you retain the fast response times as well as satisfy the asynchronous nature of today's applications.

Why asynchronous? Like most WSGI-based web frameworks, Django is synchronous. When a client requests a web page, the request reaches Django through a view and passes through various lines of code until the rendered web page is returned. As this communication waits or blocks until the process executes all this code, it is termed as synchronous.

Working Asynchronously

Chapter 25

New Django developers do not worry about creating asynchronous tasks, but I've noticed that their code eventually accumulates slow blocking tasks, such as image processing or even complex database queries, which leads to unbearably slow page loads. Ideally, they must be moved out of the request-response cycle. Page loading time is critical to user experience, and it must be optimized to avoid any delays. Another fundamental problem of this synchronous model is the handling of events that are not triggered by web requests. Even if a website does not have any visitors, it must attend to various maintenance activities. They can be scheduled at a particular time like sending a newsletter at Friday midnight, or routine background tasks such as scanning uploaded files for viruses. Some sites might offer real-time updates or push notifications through WebSockets that cannot be handled by the WSGI model. Some of the typical kinds of asynchronous tasks are: Sending a single or mass emails/SMS Calling web services Slow SQL queries Logging activity Media encoding or decoding Parsing a large corpus of text Web scraping Sending newsletters Machine learning tasks Image processing As you can see, every non-trivial Django project will need infrastructure to manage asynchronous tasks. You might also find your code running several times faster with a single process when you switch to asynchronous code (refer to the Understanding asyncio section for a dramatic example of speedup). This is because all the time you were waiting for an I/O task to complete is now better utilized running other tasks.

Pitfalls of asynchronous code Asynchronous programming might sound very compelling, but it is very difficult to master.

[ 685 ]

Working Asynchronously

Chapter 25

There are several pitfalls that you need to be aware of, such as the following: Race condition: If two or more threads of code modify the same data, the order in which they get executed can affect the final value. This race can lead to data being in an undetermined state. Starvation: Indefinite waiting by one thread due to other threads coming in. Deadlock: If a thread is waiting for a resource that another thread has locked, and vice versa at the same time, then both threads are stuck in a deadlock. Debugging challenge: It is very hard to reproduce a bug in asynchronous code due to the non-deterministic timing of a multithreaded program. Order preservation: There might be dependencies between sections of code that might not be observed when the execution order varies. In Python, it might be impossible to completely avoid such pitfalls, but we can follow some best practices to eliminate them for most practical purposes. They will be covered in the Celery best practices section.

Asynchronous patterns Let's look at various general patterns that have been used in web applications.

Endpoint callback pattern In this pattern, when a caller calls a service, it specifies an endpoint to be called when the operation is completed. This is similar to specifying callbacks in some programming languages like JavaScript. When used purely as an HTTP callback, it is called a WebHook. The process is roughly as follows: 1. The client calls a service through a channel such as REST, RPC, or UDP. It also provides its own endpoint to notify when the result becomes ready. 2. The call returns immediately. 3. When the task is completed, the service calls the defined endpoint to notify the initial sender.

[ 686 ]

Working Asynchronously

Chapter 25

Remember that the service provider or receiver must be able to access the sender. For sensitive data, there must be some form of authentication to identify the sender and encryption to protect the channel from eavesdropping. This pattern is quite popular and implemented by various web applications, such as GitHub, PayPal, Twilio, and more. These providers usually have an API to manage subscriptions to these WebHooks, unless you have a broker to perform such mediation.

Publish-subscribe pattern This pattern is a more general form of the endpoint callback pattern. Here, a broker acts as an intermediary between the actual sender and recipients. Yes, multiple recipients can subscribe to a topic i.e. a named logical group of channels published by anyone. In this case, the process of communication is as follows: 1. One or more listeners will inform a broker process that they are interested in subscribing to a topic 2. A publisher will post a message to the broker under the relevant topic 3. The broker dispatches the message to all the subscribers A broker has the advantage of fully decoupling the sender and receiver in many senses. Additionally, the broker can perform many additional tasks, such as message enrichment, transformation, or filtering. This pattern is quite scalable and, hence, popular in enterprise middleware. Celery internally uses publish/subscribe mechanisms for several of its backend transports, such as Redis for sending messages.

Polling pattern Polling, as the name suggests, involves the client periodically checking a service for any new events. This is often the least desirable means of asynchronous communication as polling increases system utilization and becomes difficult to scale. Yet, it might be the only feasible solution in a legacy system.

[ 687 ]

Working Asynchronously

Chapter 25

A polling system works as follows: 1. The client calls a service 2. The call returns immediately with new events or the status of the task 3. The client waits and repeats step two at periodic intervals There might be some degree of synchronous delay while retrieving the status of the service. The client might be blocking until the response arrives. Hence, it is sometimes referred to as busy-waiting.

Asynchronous solutions for Django The rest of this chapter will cover the following popular asynchronous systems used with Django, with somewhat different use cases. They are as listed as follows: Celery: Worker threads-based model for handling computation outside the Django process asyncio: Python built-in module for concurrently executing multiple tasks within the same thread Django Channels: Real-time message queue-like architecture to manage I/O events such as WebSockets Let's first understand the most popular and robust solution for running tasks asynchronously: Celery.

Working with Celery Celery is a feature-rich asynchronous task queue manager. Here, a task refers to a callable that, when executed, will perform the activity asynchronously. Celery is used in production by several well-known organizations including Instagram and Mozilla, for handling millions of tasks a day. While installing Celery, you will need to pick and choose various components such as a broker and result store. If you are confused, I would recommend installing Redis and skipping a result store for starters. As Redis works in-memory, if your messages are larger and need persistence, you should use RabbitMQ instead. You can follow the First Steps with Celery and Using Celery with Django topics in the Celery User Guide to get started.

[ 688 ]

Working Asynchronously

Chapter 25

In Django, Celery jobs are usually mentioned in a separate file named tasks.py within the respective app directory. Here's what a typical Celery task looks like: # tasks.py @shared_task def fetch_feed(feed_id): feed_obj = models.Feed.objects.get(id=feed_id) feed_obj.page = retrieve_page(feed_obj.feed_url) feed_obj.retrieved = timezone.now() feed_obj.save()

This task retrieves the content of an RSS feed and saves it to the database. It looks like a normal Python function (even though it will be internally wrapped by a class), except for the @shared_task decorator. This defines a Celery task. A shared task can be used by other apps within the same project. It makes the task reusable by creating independent instances of the task in each registered app. To invoke this task, you can use the delay() method, as follows: >>> from tasks import fetch_feed >>> fetch_feed.delay(feed_id=some_feed.id)

Unlike a normal function call, the execution does not jump to fetch_feed or block until the function returns. Instead, it returns immediately with an AsyncResult instance. This can be used to check the status and return value of the task. To find out how and when it is invoked, let's look at how Celery works.

How Celery works Celery can be somewhat difficult to understand due its distributed architecture. Here's a high-level diagram showing a typical Django-Celery setup:

[ 689 ]

Working Asynchronously

Chapter 25

How a typical Django Celery setup works

When a request arrives, you can trigger a Celery task while handling it. The task invocation returns immediately without blocking the process. In fact, the task has not finished execution, but a task message has entered a task queue (or one of the many possible task queues). Workers are separate processes that monitor the task queue for new tasks and actually execute them. They pick up a task message and send an acknowledgment to the queue so that the message is removed. Then they execute the task. Once completed, the process repeats, and it will try to pick up another task for execution. A worker can get blocked executing a slow task or waiting for I/O, but it does not affect the Django process by design. When the task is completed, you may configure a result store to store the results persistently. In many cases, the side effect of the task is needed and the returned result is ignored, so the result store is not required. A task can also be scheduled to run periodically using what Celery calls a Celery beat process. You can configure it to kick off tasks at certain time intervals, such as every 10 seconds or at the start of a day of the week. This is great for maintenance jobs such as backups or polling the health of a web service. Celery is well-supported, scalable, and works well with Django, but it might be too cumbersome for trivial asynchronous tasks. In such cases, I would recommend using Django Channels or RQ, a simpler Redis-based task queue. However, the best practices discussed in the next section might apply to them as well.

[ 690 ]

Working Asynchronously

Chapter 25

Celery best practices You have seen how Celery can take a lot of the heavy lifting from Django, but working with Celery is quite different from Django due to its rich feature set. There are tons of best practices mentioned in the documentation and shared in several blog posts. If you are already familiar with the concepts and want a quick checklist, check out the Celery tasks checklist at http:/​/​celerytaskschecklist.​com/​. Otherwise, read on to understand how to get the best out of Celery.

Handling failure All sorts of exceptions can happen while executing a Celery task. In the absence of a well-defined exception handling and retry mechanism, they can go undetected. Often, a job failure is temporary, such as an unresponsive API (which is beyond our control) or running out of memory. In such cases, it is better to wait and retry the task. In Celery, you can choose to retry automatically or manually. Celery makes it easy to fine-tune its automatic retry mechanism. In the following example, we specify multiple retry parameters: @shared_task(autoretry_for=(GatewayError,), retry_backoff=60, retry_kwargs={'max_retries': 5}, retry_jitter=True) def fetch_feed(feed_id): ...

The autoretry_for argument lists all the exceptions for which Celery should automatically retry. In this case, it is just the GatewayError exception. You may also mention the exception base class here to autoretry_for all exceptions. The retry_backoff argument specifies the initial wait period before the first retry, that is, 60 seconds. Each time a retry fails, the waiting period gets doubled, so the waiting period becomes 120, 240, and 360 seconds, until the maximum retry limit of 5 is reached. This technique of waiting longer and longer for a retry is called exponential backoff. This is ideal for interacting with an external server as we are giving it sufficient time to recover in case of a server overload.

[ 691 ]

Working Asynchronously

Chapter 25

A random jitter is added to avoid the problem of thundering herds. If a large number of tasks have the same retry pattern and request a resource at the same time, it might make it unusable. Hence, a random number is added to the waiting period so that such collisions do not occur. Here's an example of manually retrying in case of an exception: @shared_task(bind=True) def fetch_feed(self, feed_id): ... try: ... except (GatewayError) as exc: raise self.retry(exc=exc)

Note the bind argument to the task decorator and a new self argument to the task, which will be the task instance. If an exception occurs, you can call the self.retry method to attempt a retry manually. The exc argument is used to pass the exception information that can be used in logs. Last but not least, ensure that you log all your exceptions. You can use the standard Python logging module or the print function (which will be redirected to logs) for this. Use a tool such as Sentry to track and automate error handling.

Idempotent tasks As we saw, Celery tasks may be restarted several times, especially if you have enabled late acknowledgments. This makes it important to control the side effects of a task. Hence, Celery recommends that all tasks should be idempotent. Idempotence is a mathematical property of a function that assures that it will return the same result if invoked with the same arguments, no matter how many times you call it. You might have seen simple examples of idempotent functions in the Celery documentation itself, such as this: @app.task def add(x, y): return x + y

No matter how many times we call this function, the result of add(2, 2) is always 4.

[ 692 ]

Working Asynchronously

Chapter 25

However, it is important to understand the difference between an idempotent function and a function having no side effects (a pure or nullipotent function). The side effect of an idempotent will be the same, regardless of whether it was called once or several times. For example, a task that always places a fresh order when called is not idempotent, but a task that cancels an existing order is idempotent. Operations that only read the state of the world and do not have any side effects are nullipotent. As Celery architecture relies on tasks being idempotent, it is important to try to study all the side effects of a non-idempotent task and convert it into an idempotent task. You can do this by either checking whether the tasks have been executed previously (if it was, then abort) or storing the result in a unique location based on the arguments. An example of the latter is given in the Avoid writing to shared or global state section. Finally, call your task multiple times to test whether it leaves your system in the same state.

Avoid writing to shared or global state In a concurrent system, you can have several readers; however, the moment you have many writers accessing a shared state, you become vulnerable to the dreaded race conditions or deadlocks. It takes some planning and ingenuity to avoid all that. First, let's try to understand a race condition. Consider a Celery task A that performs some impressive image processing (such as matching your face to a celebrity). In a batch run, it picks the ten oldest uploaded images and updates a global counter. It first reads the counter's value from a database, increments it by the number of successful image matches and then overwrites the old value with the new value. Imagine that we start another identical task B in parallel to speed up the conversions. Now, if A and B reads the counter at the exact same time, they will overwrite each other's value by the end of the task, so the final value will be based on who writes in the end. In fact, the global counter's value will be highly dependent on the order in which the tasks are executed. Thus, race conditions result in invalid or corrupt data. Of course, the real issue is that the tasks are not aware of each other and a simple lock might resolve it, but locks or other synchronization primitives have problems of their own, such as starvation or deadlocks.

[ 693 ]

Working Asynchronously

Chapter 25

A practical solution will be to insert the status of each image into a table indexed with the unique identifier of an image like its hash value or file path: Image hash SHA256: b4337bc45a8f... SHA256:550cd6e1e8702...

Competed at 2018-02-09T15:15:11+05:30 2018-02-09T15:17:24+05:30

Matched image path /celeb/7112.jpg /celeb/3529.jpg

You can find the total number of successful matches by counting rows in this table. Additionally, this approach allows you to break down the successful matches by date or time. The race conditions are avoided, as we do not overwrite a global state. The only possibility of a shared state being overwritten is when two or more tasks pick up the same image for processing. Even if this happens, there is no data corruption as the result is the same and the result of the last task to finish will prevail.

Database updates without race conditions You might come across situations where updating a shared state is unavoidable. You can use row-level locks if your database supports it or Django F() objects. Notably, MySQL using MyISAM engine does not have support for row-level locks. Row-level locks are done in Django by calling select_for_update() on your QuerySet within a transaction. Consider this example: with transaction.atomic(): feed = Feed.objects.select_for_update().get(id=id) feed.html = sanitize(feed.html) feed.save()

By using select_for_update, we lock the Feed object's row until the transaction is done. If another thread or process has already locked the same row, the query will be waiting or blocked until the lock is freed. This behavior can be changed to throw an exception or skip it if locked, using the select_for_update keyword parameters.

[ 694 ]

Working Asynchronously

Chapter 25

If the operation on the field can be done within the database using SQL, it is better to use F() expressions to avoid a race condition. F() expressions avoid the need to pull the value from the database to Python memory and back. Consider the following instance: from django.db.models import F feed = Feed.objects.get(id=id) feed.subscribers = F('subscribers') + 1 feed.save()

It is only when the save() operation is performed that the increment operation is converted to an SQL expression and executed within the database. At no point is the number of feed subscribers retrieved from the database. As the database updates the new value based on the old, there is hardly a chance for a race condition between multiple threads.

Avoid passing complex objects to tasks It is easy to forget that each time we call a Celery task, the arguments get serialized before it enters the queue. Hence, it is not advisable to send a Django ORM object or any large object that might clog up the queues. There is another good reason to avoid sending a database object. Due to the asynchronous nature of execution, the data can be outdated by the time the task has begun execution. The record might have changed or even deleted. So, always pass a primary key or lookup value and retrieve the latest value of the object from the database. Celery documents refer to this as the responsibility of asserting that the world lies with the task. Ensure that your world is the present one, not the past.

Understanding asyncio asyncio is a co-operative multitasking library available in Python since version 3.6.

Celery is fantastic for running concurrent tasks out of a process, but there are certain times you will need to run multiple execution threads within the same process. If you are not familiar with async/await concepts (say from JavaScript or C#), it involves a bit of a steep learning curve. However, it is well worth your time, as it can speed up your code tremendously (unless it is completely CPU-bound). Moreover, it helps in understanding other libraries built on top of them, such as Django Channels.

[ 695 ]

Working Asynchronously

Chapter 25

All asyncio programs are driven by an event loop, which is pretty much an infinite loop that calls all registered coroutines in some order. Each coroutine operates cooperatively by yielding control to fellow coroutines at well-defined places. This is called awaiting. A coroutine is like a special function that can suspend and resume execution. It works in the same way as lightweight threads. Native coroutines use the async and await keywords, as follows: import asyncio

async def sleeper_coroutine(): await asyncio.sleep(5)

if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(sleeper_coroutine())

This is a minimal example of an event loop running one coroutine named sleeper_coroutine. When invoked, this coroutine runs until the await statement and yields control back to the event loop. This is usually where an I/O activity occurs. The control comes back to the coroutine at the same line when the activity being awaited is completed (after 5 seconds). Then, the coroutine returns or is considered completed.

asyncio versus threads If you have worked on the multithreaded code, then you might wonder, why not just use threads? There are several reasons why threads are not popular in Python. Firstly, threads need to be synchronized while accessing shared resources, or we will have race conditions. There are several types of synchronization primitives like locks but essentially, they involve waiting, which degrades performance and can cause deadlocks or starvation.

[ 696 ]

Working Asynchronously

Chapter 25

coroutine has well-defined places where execution is handed over. As a result, you

can make changes to a shared state as long as you leave it in a known state. For instance, you can retrieve a field from a database, perform calculations, and overwrite the field without worrying that another coroutine might have interrupted you in between. Secondly, coroutines are lightweight. Each coroutine needs significantly less memory than a thread. If you can run a maximum of hundreds of threads, you might be able to run tens of thousands of coroutines, given the same memory. Thread switching also takes some time (a few milliseconds). This means you might be able to run more tasks or serve more concurrent users. The downsides of coroutines is that you cannot mix blocking and non-blocking code. So once you enter the event loop, the rest of the code must be written in an asynchronous style, even the libraries you use. This might make using some older libraries with synchronous code slightly difficult.

The classic web-scraper example Let's look at an example of how we can convert synchronous code into asynchronous. We will look at a web scraper that downloads pages from a couple of URLs and measures their size. This is a popular example because it is very I/O bound and shows a significant speedup when handled concurrently.

Synchronous web-scraping The synchronous scraper only uses Python standard libraries such as urllib. It downloads the home page of three popular sites and a fourth site whose loading time can be delayed to simulate a slow connection. It prints the respective page sizes and the total running time. Here's the code for the synchronous scraper located at src/extras/sync.py: """Synchronously download a list of webpages and time it""" from urllib.request import Request, urlopen from time import time sites = [ "http://news.ycombinator.com/", "https://www.yahoo.com/", "http://www.aliexpress.com/", "http://deelay.me/5000/http://deelay.me/", ]

[ 697 ]

Working Asynchronously

Chapter 25

def find_size(url): req = Request(url) with urlopen(req) as response: page = response.read() return len(page)

def main(): for site in sites: size = find_size(site) print("Read {:8d} chars from {}".format(size, site))

if __name__ == '__main__': start_time = time() main() print("Ran in {:6.3f} secs".format(time() - start_time))

On a test laptop, this code took 17.1 seconds to run. It is the cumulative loading time of each site. Let's see how asynchronous code runs.

Asynchronous web-scraping This asyncio code requires an installation of a few Python asynchronous network libraries, such as aiohttp and aiodns. They are mentioned in the docstring. Here's the code for the asynchronous scraper at src/extras/async.py; it is structured to be as close as possible to the synchronous version so that it's easier to compare: """Asynchronously download a list of webpages and time it Dependencies: Make sure you install aiohttp pip install aiohttp aiodns """ import asyncio import aiohttp from time import time sites = [ "http://news.ycombinator.com/", "https://www.yahoo.com/", "http://www.aliexpress.com/", "http://deelay.me/5000/http://deelay.me/", ]

[ 698 ]

Working Asynchronously

Chapter 25

async def find_size(session, url): async with session.get(url) as response: page = await response.read() return len(page)

async def show_size(session, url): size = await find_size(session, url) print("Read {:8d} chars from {}".format(size, url))

async def main(loop): async with aiohttp.ClientSession() as session: tasks = [] for site in sites: tasks.append(loop.create_task(show_size(session, site))) await asyncio.wait(tasks)

if __name__ == '__main__': start_time = time() loop = asyncio.get_event_loop() loop.run_until_complete(main(loop)) print("Ran in {:6.3f} secs".format(time() - start_time))

The main function is a coroutine that triggers the creation of a separate coroutine for each website. Then, it waits until all these triggered coroutines are completed. As a best practice, the web session object is passed to avoid recreating new sessions for each page. The total running time of this program on the same test laptop is 7.5 s. This is a speedup of 2.3x on a single core. This surprising result can be better understood if we can visualize how the time was spent, as shown in the following diagram:

[ 699 ]

Working Asynchronously

Chapter 25

A simplistic representation comparing tasks in the synchronous and asynchronous scrapers

The Synchronous scraper is easy to understand. Each task is waiting for the previous task to complete. Each task needs very little CPU time and the majority of the time is spent waiting for the data to arrive from the network. As a result, the tasks cascade sequentially like a waterfall. On the other hand, the Asynchronous scraper starts the first task and, as soon as it starts waiting for I/O, it switches to the next task. The CPU is hardly idle as the execution goes back to the event loop as soon as the waiting starts. Eventually, the I/O completes in the same amount of time, but due to the multiplexing of activity, the overall time taken is drastically reduced. In fact, the asynchronous code can be sped up further. The standard asyncio event loop is written in pure Python and provided as a reference implementation. You can consider faster implementations such as uvloop to speed things up further.

[ 700 ]

Working Asynchronously

Chapter 25

Concurrency is not parallelism Concurrency is the ability to perform other tasks while you are waiting on the current task. Imagine that you are cooking a lot of dishes for some guests. While waiting for something to cook, you are free to do other things like peeling onions or cutting vegetables. To make an analogy in the world of superheroes, a superhero might battle several bad guys at one place because most would be either recovering from a blow, arriving (or ahem waiting for their turn), which leaves our hero to deliver blows one at a time. Parallelism is when two or more execution engines are performing a task. Continuing on our analogy, this is when two or more superheroes battle enemies as a team. This is not only a great cinema franchise opportunity, but also more productive than a single hero working at maximum efficiency. It is very easy to confuse concurrency and parallelism because they can happen at the same time. You could be concurrently running tasks without parallelism or vice versa, but they refer to two different things. Concurrency is a way of structuring your programs, while parallelism refers to how it is executed. Due to the global interpreter lock (GIL), we cannot run more than one thread of the Python interpreter (to be specific, the standard CPython interpreter) at a time, even in multicore systems. This limits the amount of parallelism that we can achieve with a single instance of the Python process. Optimal usage of your computing resources requires both concurrency and parallelism. Concurrency will help you avoid blocking the processor core while waiting for, say, I/O events, while parallelism will help to distribute work among all the available cores. In both cases, you are not executing synchronously, that is, waiting for a task to finish before moving on to another task. Asynchronous systems might seem to be the most optimal; however, they are harder to build and reason about.

Entering Channels Django Channels was originally created to solve the problem of handling asynchronous communication protocols, such as WebSockets, for example. More and more web applications were providing real-time capabilities such as chat and push notifications. Various hacks were created to make Django support requirements including running separate socket servers or proxy servers.

[ 701 ]

Working Asynchronously

Chapter 25

Channels is an official Django project, not just for handling WebSockets and other forms of bi-directional communication but also for running background tasks asynchronously. As at the time of writing, Django Channels 2 is out, which is a complete rewrite based on Python 3's async/await-based coroutines. Here's a simplified block diagram of a typical Channels setup:

How a typical Django Channels infrastructure works

A client, such as a web browser, sends both HTTP/HTTPS and WebSocket traffic to an Asynchronous Server Gateway Interface (ASGI) server such as Daphene. Like WSGI, the ASGI specification is a common way for application servers and applications to interact with each other asynchronously. Like a typical Django application, HTTP traffic is handled synchronously, that is, when the browser sends a request, it waits until it is routed to Django and a response is sent back. However, it gets a lot more interesting when WebSocket traffic happens, because it can be triggered from either direction. Once a WebSocket connection is established, a browser can send or receive messages. A sent message reaches the protocol type router that determines the next routing handler based on its transport protocol. Hence, you can define a router for HTTP and another for WebSocket messages. These routers are very similar to Django's URL mappers, but map the incoming messages to a consumer (rather than a view). A consumer is like an event handler that reacts to events. It can also send messages back to the browser, thereby containing the logic for a fully bi-directional communication.

[ 702 ]

Working Asynchronously

Chapter 25

A consumer is a class whose methods you may choose to write either as normal Python functions (synchronous) or as awaitables (asynchronous). An asynchronous code should not mix with synchronous code, so there are conversion functions to convert from async to sync and back. Remember that the Django parts are synchronous. A consumer is, in fact, a valid ASGI application. So far, we have not used the Channel layer. Ironically, you can write Channel applications without using Channels! However, they are not particularly useful as there is no easy communication path between application instances, other than polling a database. Channels provide exactly that, a fast point-to-point and broadcast messaging between application instances. A channel is like a pipe. A sender sends a message to this pipe from one end, and it reaches a listener at the other end. A group defines a group of Channels who are all listening to a topic. Every consumer listens to their own autogenerated channel accessed by its self.channel_name attribute. In addition to transports, you can trigger a consumer listening to a channel by sending a message, thereby starting a background task. This works as a very quick and simple background worker system.

Listening to notifications with WebSockets Instead of the usual chat example, let's look at an example better suited to a social network to illustrate Channels—a notification app. The app will detect whenever a certain type of model is saved and push a notification to all clients (that is, browsers of all the connected users) in real time. Assuming that Channels is properly installed and configured, we need to define all the protocol type routes in the routing.py file, as follows: from channels.routing import ProtocolTypeRouter, URLRouter from django.urls import path from notifier.consumers import NotificationConsumer

application = ProtocolTypeRouter({ "websocket": URLRouter([ path("notifications/", NotificationConsumer), ]), })

[ 703 ]

Working Asynchronously

Chapter 25

HTTP requests are sent to Django, by default. This leads us to the code of the consumer, residing within the notification app itself as consumers.py: from channels.generic.websocket import AsyncJsonWebsocketConsumer

class NotificationConsumer(AsyncJsonWebsocketConsumer): async def connect(self): await self.accept() await self.channel_layer.group_add("gossip", self.channel_name) async def disconnect(self, close_code): await self.channel_layer.group_discard("gossip", self.channel_name) async def name_gossip(self, event): await self.send_json(event)

For convenience, we are using a generic consumer class called AsyncJsonWebsocketConsumer, which handles WebSocket communication by translating to and from the JSON format. The connect method simply accepts a connection and adds its channel to the gossip Channel group. Now, any message posted to this group will invoke an appropriately named class method of this consumer. We are only interested in messages that have the name.gossip type; hence, we have created a method called name_gossip (dots are translated into underscores). This method simply sends the given event object to the WebSocket, which is received by the browser. The disconnect method ensures that the consumer's Channel is removed from the group when the connection is closed. Thus, we will have only active channels in the group. The only remaining bit of the puzzle is what triggers the event. We have the following code in the signals.py file of the app: from from from from from

.post.models import Post django.db.models.signals import pre_save django.dispatch import receiver asgiref.sync import async_to_sync channels.layers import get_channel_layer

[ 704 ]

Working Asynchronously

Chapter 25

@receiver(pre_save, sender=Post) def notify_post_save(sender, **kwargs): if "instance" in kwargs: instance = kwargs["instance"] # check if it is a new post ... channel_layer = get_channel_layer() async_to_sync(channel_layer.group_send)( "gossip", {"type": "name.gossip", "event": "New Post", "sender": instance.posted_by.get_full_name(), "message": instance.message})

We are adding a hook to be called whenever a Post object (it can be any object for that matter) is saved. As we are only interested in new posts, we check and ignore the edits of the existing posts. Before we send anything to a channel, we need to retrieve the channel_layer. Then, we need to use the group_send method to send the message to the gossip group. However, this is an asynchronous method, and we are in the Django world, so it is happening synchronously. Hence, we wrap the call using an async_to_sync converter, making it essentially block until the async function returns. As you might have noted, Channels uses the publish-subscribe pattern. The design of channels deliberately avoids waiting for an event and, hence, prevents deadlocks. By basing on asyncio, we can build true asynchronous applications with Django.

Differences from Celery With the ability to run background tasks using workers, you might naturally be confused if Channels can replace Celery. There are primarily two major differences: message delivery guarantees and task statuses. Channels, currently implemented with a Redis backend, provide an at best one-off guarantee, while Celery provides an at least one-off guarantee. This essentially means that Celery will retry when a delivery fails until it receives a successful acknowledgment. In the case of Channels, it is pretty much fire-and-forget. Secondly, Channels does not provide information on the status of a task out of the box. We need to build such functionality ourselves, for instance by updating the database. Celery tasks status can be queried and persisted.

[ 705 ]

Working Asynchronously

Chapter 25

To sum up, you can use Channels instead of Celery for some less critical use cases. However, for a more robust and proven solution, you should rely on Celery.

Summary In this chapter, we looked at various ways to support asynchronous execution in Django. They provide powerful abstractions on top of Django to create applications that can support push notifications, display the progress of a slow task, communicate with other users, or run background tasks. Traditionally, Celery has been the tool of choice for asynchronous activities. However, Channels provide a lighter and more tightly integrated solution. Both have their uses and can be used in the same project. Use the right tool for the job! In the next chapter, we will look at what RESTful APIs means and how we can implement them in Django using current best practices.

[ 706 ]

26 Creating APIs In this chapter, we will discuss the following topics: RESTful API API design Django Rest framework API Patterns So far, we have been designing Django applications to be consumed by humans. But many of our applications are also consumed by other applications, that is, machine to machine. A well-designed API makes it easier for programmers to write code that uses it. In this chapter, we will be referring to Representational state transfer (REST) web APIs whenever we use the term APIs, as it is popularly implied. These APIs have become a popular means not just for accessing web application functionality, but also for mashing up and creating entirely new applications.

RESTful API Most applications and popular websites provide a REST application programming interface (API) these days. Amazon, Netflix, Twillio, and thousands of companies have a public-facing interface that has become a significant part of their business growth. A RESTful API is a web service API that adheres to the REST architectural properties. Due to its simplicity and flexibility for a variety of use cases such as mobile applications, it has become a de facto standard in the industry for programmatic interfaces.

Creating APIs

Chapter 26

There are six architectural constraints of a pure RESTful system, and these are, as follows: Client-server: Mandates that client and server must be separate and allowed to evolve independently Stateless: Requires REST calls to be stateless, that is, client context is not stored on the server but at the client Cacheable: Specifies that responses must define themselves to be cacheable or not, which can improve scalability and performance Layered system: Forms a hierarchy that helps manage complexity and improve scalability Code on demand: Allows for code or applets to be sent by servers to clients Uniform Interface: Is a fundamental set of constraints that decouples the architecture, such as resources and self-descriptive messages However, most modern APIs are not purely RESTful because they break one or more of these constraints (usually the Uniform Interface). However, they might still be called REST APIs. Practically, most adhere to a few architectural concepts, such as these: Resources: Any object, data or service accessible by a Uniform Resource Identifiers (URI). This can be a single object (say a User) or a collection (say Users). Usually, they refer to a noun rather than a verb. Request operations: Operations on resources generally done using standard HTTP operations such as GET, PUT, POST, OPTIONS, and DELETE. They follow the same rules as well, such as GET is nullipotent (has no side effects) and PUT/DELETE is idempotent (the same result no matter how many times it gets executed). Error codes: REST APIs use standard HTTP error codes such as 200 (success), 300 (redirection), and 400 (user error). Hypermedia: Responses will usually contain hyperlinks or URIs to other actions and resources for flexibility and discoverability. For instance, use hyperlinks for pagination or nested data structures. My recommendation will make your API as easy to use as possible rather than to strictly follow the pure REST constraints. Many well-known and popular APIs violate some of them. If a REST-ish API design is cleaner than otherwise, go for it!

[ 708 ]

Creating APIs

Chapter 26

API design We do not have a single standard for a REST API. However, over time, many welldesigned APIs by companies such as Stripe, GitHub, and Trello have become standards around which web APIs are now being designed. Here, we shall cover some best practices in addition to the architectural principles we outlined earlier.

Versioning An API is like a contract between a client and server. If either interface changes, typically on the server side, the contract fails. However, APIs need to evolve, as new features get added and old ones get deprecated. Hence, the API versioning is a key design decision taken early on in an API lifecycle. There are several popular API versioning implementations: URI versioning: Prefixing the URI with the version number, such as http:/​/​example.​com/​v3/​superheroes/​3 . This is a popular method but violates the principle that each resource has a unique URI across versions. Query string versioning: Appending the URI with a query string specifying the version, such as http:/​/​example.​com/​superheroes/​3? version=​3 . Technically, the URI is the same across versions, but such responses are not cached in older web proxies, thereby degrading performance. Custom header versioning: Including a custom header in your requests; take the following for instance: GET /superheroes/3 HTTP/1.1 Host: example.com Accept: application/json api-version: 3

While this might be closer to REST principles and cleaner, it can be harder to test in some web clients, like browsers. Custom Headers are outside specs and might cause latent issues that can be hard to debug.

[ 709 ]

Creating APIs

Chapter 26

Media type versioning: Use the Accept header to specify a custom media type that explicitly mentions the version; consider this for instance: GET /superheroes/3 HTTP/1.1 Host: example.com Accept: application/vnd.superhero-api.v3+json

While this may also have testing issues, like custom headers, it honors the standard. This might be the purest REST versioning model. There are other design decisions to make too, such as which versioning scheme should be followed? Should it be a simple incrementing integer (as in the preceding examples), a semantic version (like Facebook), or the release date (like Twilio)? It is quite similar to a product versioning exercise. Backward compatibility is also an important API lifecycle decision. How many older versions to keep? What determines a minor or major version change? How to deprecate older versions? It is best to have a clearly communicated policy that is followed consistently.

Django Rest framework Creating your website's API might seem trivial using the services pattern that we learned so far. However, real-world APIs need so much more functionality, such as web browsable documentation, authentication, serialization, and throttling, that you are better off using a toolkit such as Django Rest framework (DRF). DRF is the most popular API toolkit for Django. It fits well with the Django architecture and reuses several familiar concepts such as generic views and model forms. Out of the box, the API is accessible and usable with a normal web browser, which makes testing and finding documentation easier for developers.

Improving the Public Posts API Recall the services pattern example where we created a service to retrieve all the latest public posts? Now we shall reimplement it using the features provided by the DRF.

[ 710 ]

Creating APIs

Chapter 26

First, install DRF and add it to your INSTALLED_APPS. Then, mention your permission model in settings.py: # Django Rest Framework settings REST_FRAMEWORK = { # Allow unauthenticated access to public content 'DEFAULT_PERMISSION_CLASSES': [ 'rest_framework.permissions.AllowAny' ] }

Even though we are allowing unrestricted access (AllowAny) here, it is strongly recommended to choose the most restricted access policy to secure your API. DRF allows us to choose from a wide variety of API access permission policies, such as allowing only authenticated users (IsAuthenticated) or allowing unauthenticated users read-only access (DjangoModelPermissionsOrAnonReadOnly), and more. More fine-grained object level permissions can also be defined. Since we already have the Post model and model manager for public posts defined earlier, we shall create the Post serializer. Serializers are used for converting structured objects, such as model instances or QuerySets, into formats like JSON or XML that can be sent over the wire. They also perform the reverse function of deserialization, that is, parsing a JSON or XML back into a structured object. Create a new file called viewschapter/serializers.py with the following content: from rest_framework import serializers from posts import models

class PostSerializer(serializers.ModelSerializer): class Meta: model = models.Post fields = ("posted_by_id", "message")

We are declaratively defining the serializers class by referring to the model class and the fields, which need to be serialized or deserialized. Note how this looks similar to defining a ModelForm.

[ 711 ]

Creating APIs

Chapter 26

This is intentional. Such as an HTML-based website needs forms to validate user input, a web API needs a deserializer to validate the data submitted to the API. Just as forms mapped to models are called ModelForms, serializers mapped to models are called ModelSerializers. Next, we define our API view in a separate file called viewschapter/apiviews.py: from rest_framework.views import APIView from rest_framework.response import Response from posts import models from .serializers import PostSerializer

class PublicPostList(APIView): """ Return the most recent public posts by all users """ def get(self, request): msgs = models.Post.objects.public_posts()[:5] data = PostSerializer(msgs, many=True).data return Response(data)

APIView class methods use different parameters and return types compared to Django's View class. It takes REST framework's Request instances, rather than Django's HttpRequest instances. It also returns REST framework's Response instances instead of Django's HttpResponse instances. However, it can be used just like a View class.

Finally, we wire this into our app's viewschapter/urls.py: path('api/public/', apiviews.PublicPostList.as_view(), name="api_public"),

Now, if you visit the http://127.0.0.1:8000/api/public/ API endpoint on your browser, you will see this awesome page:

[ 712 ]

Creating APIs

Chapter 26

Compare this to the earlier chapter's view that returned just a bare JSON string. We can see the name of this API endpoint and its description (from the APIView class docstring), the request headers, and the JSON payload itself (with syntax highlighting).

Hiding the IDs The API looks great, except for the security risk of exposing the user model's primary key publicly. Thankfully, the serializers can be changed to add fields that are not present in the model, as the following code demonstrates: class PostSerializer(serializers.ModelSerializer): posted_by = serializers.SerializerMethodField() def get_posted_by(self, obj): return obj.posted_by.username class Meta: model = models.Post fields = ("posted_by", "message",)

The SerializerMethodField is a read-only field that gets its value from a class method. By default, this is the method named get_.

[ 713 ]

Creating APIs

Chapter 26

Now, the API returns posts with the usernames instead of the user's primary key, as the following screenshot shows:

If you are a REST purist, you might point out that instead of a username, we can use hyperlinks to the User resource. You may want to implement this if your users are comfortable with sharing their details on a public API.

API patterns This section covers some familiar design problems while working with APIs.

Pattern – human browsable interface Problem: Visiting an API in a browser is a jarring experience, leading to poor adoption. Solution: Use the opportunity to provide a human browsable interface to your API.

[ 714 ]

Creating APIs

Chapter 26

Problem details Even though APIs are designed to be consumed by code, the initial interaction is typically by a human. A working implementation might respond with correct results if the right parameters are passed, but without proper documentation, it can be unusable. Under-documented APIs can reduce collaboration by different teams with your application. Often, required resources such as conceptual overviews and getting started guides are not found, leading to a frustrating developer experience. Finally, since most web APIs are initially accessed using web browsers, an ability to interact with the API within the documentation itself is very useful. Even if the documented behavior differs from the code, the ability to try and verify the behavior within the browser helps in testing.

Solution details DRF has built-in support for creating a human browsable interface that addresses several problems mentioned in this pattern. Visiting an API endpoint using a browser generates a documentation of the API endpoint with the supported HTTP operations and an ability to interact with them. Your API documentation can be made more comprehensive and interactive using Swagger, or using DRF's own coreapi tool. Swagger has the ability to find all the API endpoints of your application without access to its source code. It can also be used for testing the endpoints by sending requests and responses. Alternatively, you can use coreapi quite easily by plugging a line to your urls.py; consider the following by way of an example: from rest_framework.documentation import include_docs_urls urlpatterns = [ path('api-docs/', include_docs_urls(title='Superbook API')), ]

[ 715 ]

Creating APIs

Chapter 26

If you visit the preceding location in your browser, you will see the following readyto-use API documentation:

Note how the API documentation includes code examples in Python (and other languages). Some best practices to follow while creating an API documentation are as listed: Easy and quick onboarding: Make it easy for developers to get up and running with ready-to-run examples and tutorials. Ideally, it should not take a developer more than five minutes to understand your API and start using it. Interactive sandbox: Give your interactive documentation demo user credentials and some representative sample data to work with, rather than keeping it empty. Go beyond endpoints: Ensure that you cover essential topics such as how to obtain authentication tokens or pricing, as well as high-level concepts. Good API documentation is crucial for its adoption and can even overcome a poorly designed API, so it is worth putting your time and effort into it.

Pattern – Infinite Scrolling Problem: Users consume limited content on paginated views Solution: Engage users longer using pages with Infinite Scrolling

[ 716 ]

Creating APIs

Chapter 26

Problem details Casual visitors to your website have a great appetite for consuming lots of content, be it a social news feed or trendy clothing. However, they find clicking on the link to cross over to the next page quite annoying. Mobile users might find the experience even more jarring as they find scrolling through a larger list more intuitive.

Solution details Traditionally, a page containing a lot of data was paginated to reduce page loading time and thereby improve the user experience. Then, Asynchronous JavaScript And XML (AJAX) technologies gave browsers the ability to asynchronously load content. Thus, the Infinite Scrolling design pattern was born, where by new content was continually added as the user reached the bottom of the page. This is a very common technique in social media sites such as Facebook or Twitter to increase user engagement with minimal interaction. However, not all users consider Infinite Scroll pages to be an improvement. They can get disoriented when they look for specific content in a page several screens long. Poor implementations can break the Back button functionality of the browser when trying to return to the same place on the previous page. The recommended solution is as follows: 1. Use JavaScript to listen to the scroll event until it reaches a certain mark. 2. When the mark is reached, the next page link is asynchronously requested (AJAX). 3. The link is handled by a Django service or REST API. It returns the appropriate page and next page link. 4. The new content is appended to the page. 5. Optionally, use the browser's pushState API to update the URL to the last loaded page. Essentially, we need an AJAX backend provided by Django that supplies the appropriate page of content. A suitable generic view for this case might be the ListView, with the paginate_by parameter set to the number of objects per page.

[ 717 ]

Creating APIs

Chapter 26

Infinite Scroll is a very impressive trick, which, when executed well, can feel literally seamless to users. However, it requires careful user testing to understand whether it is appropriate to the content being viewed. For example, Google uses infinite scrolling for Google Images searches but uses pagination for regular searches, so it might not be the best technique for all scenarios.

Summary In this chapter, we studied the conceptual underpinnings of a RESTful API and why we do not have to strictly adhere to all of it. We also looked at the DRF and a very simple example of an API endpoint created using it. In the next chapter, we will take a look at a systematic approach to working with a legacy Django code base and how we can enhance it to meet evolving client needs.

[ 718 ]

27 Production-Ready In this chapter, we will discuss the following topics: Picking a web stack Hosting approaches Deployment tools Monitoring Performance tips So, you have developed and tested a fully functional web application in Django. Deploying this application can involve a diverse set of activities from choosing your hosting provider to performing installations. Even more challenging could be the tasks of maintaining a production site so it works without interruption and handling unexpected bursts in traffic. The discipline of system administration is vast. Hence, this chapter will cover a lot of ground. However, given the limited space, we will attempt to familiarize you with the various aspects of building a production environment.

The production environment Although most of us intuitively understand what a production environment is, it is worthwhile clarifying what it really means. A production environment is simply one where end users use your application. It should be available, resilient, secure, responsive, and must have abundant capacity for current (and future) needs.

Production-Ready

Chapter 27

Unlike a development environment, the chance of real business damage due to any issues in a production environment is high. Hence, before moving to production, the code is moved to various testing and acceptance environments in order to get rid of as many bugs as possible. For easy traceability, every change made to the production environment must be tracked, documented, and made accessible to everyone in the team. As an upshot, there must be no development performed directly on the production environment. In fact, there is no need to install development tools, such as a compiler or debugger, in production. The presence of any unneeded software increases the attack surface of your site and could pose a security risk. Most web applications are deployed on sites with extremely low downtime, for example, large data centers are at five nines, that is, 99.999 percent, uptime. By designing for failure, even if an internal component fails, there is enough redundancy to prevent the entire system crashing. This concept of avoiding a single point of failure (SPOF) can be applied at every level, hardware or software. Hence, it is a crucial collection of software you choose to run in your production environment.

Choosing a web stack So far, we have not discussed the stack on which your application will be running. Even though we are talking about it at the very end of this book, it is best not to postpone such decisions to the later stages of the application lifecycle. Ideally, your development environment must be as close as possible to the production environment to avoid the but it works on my machine situation. By a web stack, we refer to the set of technologies that are used to build a web application. It is usually depicted as a series of components, such as OS, database, and web server, all piled on top of one another. Hence, it is referred to as a stack. We will mainly focus on open source solutions here because they are widely used. However, various commercial applications can also be used if they are more suited to your needs.

[ 720 ]

Production-Ready

Chapter 27

Components of a stack A production Django web stack is built using several kinds of application (or layers, depending on your terminology). While constructing your web stack, some of the choices you might need to make are as follows: Which OS and distribution? For example, Debian, Red Hat, or OpenBSD. Which WSGI server? For example, Gunicorn or uWSGI. Which web server? For example, Apache or Nginx. Which database? For example, PostgreSQL, MySQL, or Redis. Which caching system? For example, Memcached or Redis. Which process control and monitoring system? For example, Upstart, Systemd, or Supervisord. How to store static media? For example, Amazon S3 or CloudFront There could be several more, and these choices are not mutually exclusive either. Some use several of these applications in tandem. For example, username availability might be looked up on Redis, while the primary database might be PostgreSQL. There is no one size fits all answer when it comes to selecting your stack. Different components have different strengths and weaknesses. Choose them only after careful consideration and testing. For instance, you might have heard that Nginx is a popular choice for a web server, but you might actually need Apache's rich ecosystem of modules or options. Sometimes, the selection of the stack is based on various non-technical reasons. Your organization might have standardized on a particular operating system, say, Debian for all its servers, or your cloud hosting provider might support only a limited set of stacks. Hence, how you choose to host your Django application is one of the key factors in determining your production setup.

Virtual machines or Docker Most of us are familiar with using virtual machines either in development or in production. They isolate your application (guest machine) from the underlying infrastructure (host machine). Container technologies such as Docker are increasingly being used for cloud deployments, either complementing, or replacing virtual machines.

[ 721 ]

Production-Ready

Chapter 27

Containers are a means to create multiple user-space instances over the same kernel. Unlike virtual machines, containers avoid the need to start, and run separate guest operating systems. Typically, each container packages an application and its dependencies in a user-space instance separate from other containers. Unlike virtual machines, they do not have a separate instance of the operating system, making them lighter, and faster to start or stop. Docker has become the containerization technology of choice with a large ecosystem and wide support among cloud vendors. Docker images are created from a binary image called base image or automatically built from a script called a Dockerfile. This helps you recreate the same environment in production for development or testing purposes, thus ending the infamous excuse but it worked in my machine.

Microservices The most common design pattern using Docker is breaking down applications and services into microservices. The advantage is that individual microservices can be developed and deployed independently while being more elastic and resilient in demanding situations. Hence, containerization technologies such as Docker is a natural fit due to its minimal overhead and application-level isolation. The following is a simplistic example of a Django web application implemented as microservice using containers:

Django application flow when deployed as distinct containers

[ 722 ]

Production-Ready

Chapter 27

This single microservice is composed of three containers with separate logical components: Nginx container (web server), Gunicorn/Django container (web application), and PostgreSQL container (database). Each container is instantiated from a Docker image, which may be built using a Dockerfile. Docker containers have an ephemeral file system, so persistent data is managed by explicitly creating a volume. Volumes can be used to share data between containers. In this case, the static files of the Django project can be shared to the Nginx container to serve them directly. As you can imagine, most real-world applications will be composed of multiple Microservices and each of them would require multiple containers. If you run them on multiple servers, how would you deploy these containers across them? How can you scale individual microservices up or down? Kubernetes is the most widely recommended solution for managing such container clusters. Although we have covered containers in this section at a very high level, there are many implementation details, such as deployment patterns, which could not be covered here, as they can be a book by itself. Containers and orchestration tools have become an important part of modern web application development by making radically easier-to-manage application environments.

Hosting When it comes to hosting, you will need to be sure whether to go for a hosting platform such as Heroku or not. If you do not know much about managing a server or do not have anyone with that knowledge in your team, then a hosting platform is a convenient option.

Platform as a service A Platform as a Service (PaaS) is defined as a cloud service where the solution stack is already provided and managed for you. Popular platforms for Django hosting include Heroku, PythonAnywhere, and Google App Engine. In most cases, deploying a Django application should be as simple as selecting the services or components of your stack and pushing out your source code. You do not have to perform any system administration or setup yourself. The platform is entirely managed.

[ 723 ]

Production-Ready

Chapter 27

Like most cloud services, the infrastructure can also scale on demand. If you need an additional database server or more RAM on a server, it can be easily provisioned from a web interface or the command line. The pricing is primarily based on your usage. The bottom line with such hosting platforms is that they are very easy to set up and ideal for smaller projects. They tend to be more expensive as your user base grows. Another downside is that your application might get tied to a platform or become difficult to port. For instance, Google App Engine is used to support only a nonrelational database, which means you need to use django-nonrel, a fork of Django. This limitation is now somewhat mitigated with Google Cloud SQL.

Virtual private servers A virtual private server (VPS) is a virtual machine hosted in a shared environment. From the developer's perspective, it would seem like a dedicated machine (hence, the word private) preloaded with an operating system. You will need to install and set up the entire stack yourself, though many VPS providers such as WebFaction and DigitalOcean offer easier Django setups. If you are a beginner and can spare some time, I highly recommend this approach. You will be given root access, and you can build the entire stack yourself. You will not only understand how various pieces of the stack come together but also have full control in fine-tuning each individual component. Compared to a PaaS, a VPS might work out to be more value for money, especially for high-traffic sites. You might be able to run several sites from the same server as well.

Serverless Imagine that you need to host an infrequently used service, but paying for a dedicated server that is always up and running is proving to be costly or inefficient to maintain. Serverless architectures might be what you are looking for. The name serverless is a misnomer since all client requests are indeed handled by servers, which are dynamically provisioned for the lifetime of the request.

[ 724 ]

Production-Ready

Chapter 27

A more appropriate term would be Function as a Service (FaaS), as these platforms support execution of an application logic like a small Python function but does not store any state. Building an application composed of such functions would be quite similar to the microservices architecture discussed earlier. Typically, you only pay for the milliseconds of server time that a serverless application uses, which makes it much cheaper than dedicated servers. Scaling is automatically handled, so there is no additional effort needed to handle massive spikes in traffic. Last but not the least, there is no headache of having to set up and maintain server infrastructure. Django might not sound like it would work in such an environment, but Zappa makes it easy to deploy Django applications (in fact, any WSGI compatible application) on a serverless platform such as AWS Lambda with minimal changes. This opens up the possibility of enjoying all the advantages of serverless while using Django.

Other hosting approaches Even though hosting on a platform or VPS are by far the two most popular hosting options, there are plenty of other options. If you are interested in maximizing performance, you can opt for a bare metal server with collocation from providers, such as Rackspace. On the lighter end of the hosting spectrum, you can save the cost by hosting multiple applications within Docker containers. Docker is a tool to package your application and dependencies in a virtual container. Compared to traditional virtual machines, a Docker container starts up faster and has minimal overheads (since there is no bundled operating system or hypervisor). Docker is ideal for hosting micro services-based applications. It is becoming as ubiquitous as virtualization with almost every PaaS and VPS provider supporting them. It is also a great development platform since Docker containers encapsulate the entire application state and can be directly deployed to production.

[ 725 ]

Production-Ready

Chapter 27

Deployment tools Once you have zeroed in on your hosting solution, there could be several steps in your deployment process, from running regression tests to spawning background services. The key to a successful deployment process is automation. Since deploying applications involves a series of well-defined steps, it can be rightly approached as a programming problem. Once you have an automated deployment in place, you do not have to worry about deployments for fear of missing a step. In fact, deployments should be painless and as frequent as required. For example, the Facebook team can release code to production several times in a day. Considering Facebook's enormous user base and code base, this is an impressive feat, yet, it becomes necessary as emergency bug fixes and patches need to be deployed as soon as possible. A good deployment process is also idempotent. In other words, even if you accidentally run the deployment tool twice, the actions should not be executed twice (or rather it should leave it in the same state). Let's take a look at some of the popular tools for deploying Django applications.

Fabric Fabric is favored among Python web developers for its simplicity and ease of use. It expects a file named fabfile.py that defines all the actions (for deployment or otherwise) in your project. Each of these actions can be a local or remote shell command. The remote host is connected via SSH. The key strength of Fabric is its ability to run commands on a set of remote hosts. For instance, you can define a web group that contains the hostnames of all web servers in production. You can run a Fabric action only against these web servers by specifying the web group name on the command line.

To illustrate the tasks involved in deploying a site using Fabric, let's take a look at a typical deployment scenario.

[ 726 ]

Production-Ready

Chapter 27

Typical deployment steps Imagine that you have a medium-sized web application deployed on a single web server. Git has been chosen as the version control and collaboration tool. A central repository that is shared with all users has been created in the form of a bare Git tree. Let's assume that your production server has been fully set up. When you run your Fabric deployment command, say, fab deploy, the following scripted sequence of actions take place: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Runs all tests locally Commits all local changes to Git Pushes to a remote central Git repository Resolves merge conflicts, if any Collects the static files (CSS, images) Copies the static files to the static file server At the remote host, pulls changes from a central Git repository At the remote host, runs (database) migrations At the remote host, touches app.wsgi to restart WSGI server

The entire process is automatic and should be completed in a few seconds. By default, if any step fails, then the deployment gets aborted. Though not explicitly mentioned, there would be checks to ensure that the process is idempotent. Fabric is not yet compatible with Python 3, though the developers are in the process of porting it. In the meantime, you can run Fabric in a Python 2.x virtual environment or check out similar tools, such as PyInvoke.

Configuration management Managing multiple servers in different states can be hard with Fabric. Configuration management tools such as Chef, Puppet, or Ansible try to bring a server to a certain desired state.

[ 727 ]

Production-Ready

Chapter 27

Unlike Fabric, which requires the deployment process to be specified in an imperative manner, these configuration-management tools are declarative. You just need to define the final state you want the server to be in, and it will figure out how to get there. For example, if you want to ensure that the Nginx service is running at startup on all your web servers, then you will need to define a server state having the Nginx service both running and starting on boot. On the other hand, with Fabric, you will need to specify the exact steps to install and configure Nginx to reach such a state. One of the most important advantages of configuration-management tools is that they are idempotent by default. Your servers can go from an unknown state to a known state, resulting in an easier server configuration management and reliable deployment. Among configuration-management tools, Chef, and Puppet enjoy wide popularity since they were one of the earliest tools in this category. However, their roots in Ruby can make them look a bit unfamiliar to the Python programmer. For such folks, we have Salt and Ansible as excellent alternatives. Configuration-management tools have a considerable learning curve compared to simpler tools, such as Fabric. However, they are essential tools for creating reliable production environments and are certainly worth learning.

Monitoring Even a medium-sized website can be extremely complex. Django might be one of the hundreds of applications and services running and interacting with each other. In the same way that the heartbeat and other vital signs can be constantly monitored to assess the health of the human body, so are various metrics collected, analyzed, and presented in most production systems. While logging keeps track of various events, such as the arrival of a web request or an exception, monitoring usually refers to collecting key information periodically, such as memory utilization, or network latency. However, differences get blurred at the application level, for example, while monitoring database query performance, which might very well be collected from logs. Monitoring also helps with the early detection of problems. Unusual patterns, such as spikes or a gradually increasing load, can be signs of bigger underlying problems, such as memory leak. A good monitoring system can alert site owners of problems before they happen.

[ 728 ]

Production-Ready

Chapter 27

Monitoring tools usually need a backend service (sometimes called agents) to collect the statistics and frontend service to display dashboards or generate reports. Popular data collection backends include StatsD and Monit. This data can be passed to frontend tools, such as Graphite. There are several hosted monitoring tools, such as New Relic and Status.io, which are easier to set up and use. Measuring performance is another important role of monitoring. As we will soon see in a later section, any proposed optimization must be carefully measured and monitored before getting implemented.

Improving Performance Performance is a feature. Studies show how slow sites have an adverse effect on users, and therefore revenue. For instance, tests at Amazon in 2007 revealed that for every 100 ms increase in load time of amazon.com, the sales decreased by 1 percent. Reassuringly, several high-performance web applications such as Disqus and Instagram have been built on Django. At Disqus, in 2013, they could handle 1.5 million concurrently connected users, 45,000 new connections per second, 165,000 messages per second, with less than 0.2 seconds latency end-to-end. The key to improving performance is finding where the bottlenecks are. Rather than relying on guesswork, it is always recommended that you measure and profile your application to identify these performance bottlenecks. As Lord Kelvin would say: "If you can't measure it, you can't improve it." In most web applications, the bottlenecks are likely to be at the browser or the database end rather than within Django. However, to the user, the entire application needs to be responsive. Let's take a look at some of the ways to improve the performance of a Django application. Due to widely differing techniques, the tips are split into two parts: frontend and backend.

[ 729 ]

Production-Ready

Chapter 27

Frontend performance Django programmers might quickly overlook frontend performance because it deals with understanding how the client side, usually a browser, works. However, let's quote Steve Souders' study of Alexa-ranked top 10 websites: "80-90% of the end-user response time is spent on the frontend. Start there." A good starting point for frontend optimization would be to check your site with Google page speed or Yahoo! YSlow (commonly used as browser plugins). These tools will rate your site and recommend various best practices, such as minimizing the number of HTTP requests or gzipping the content. As a best practice, your static assets, such as images, stylesheets, and JavaScript files, must not be served through Django. Rather a static file server, cloud storages such as Amazon S3, or a content delivery network (CDN) should serve them for better performance. Even then, Django can help you improve frontend performance in a number of ways: Cache infinitely with CachedStaticFilesStorage: The fastest way to load static assets is to leverage the browser cache. By setting a long caching time, you can avoid re-downloading the same asset again and again. However, the challenge is to know when not to use the cache when the content changes. CachedStaticFilesStorage class solves this elegantly by appending the asset's MD5 hash to its filename. This way, you can extend the TTL of the cache for these files infinitely. To use this, set the CACHES setting named staticfiles to CachedStaticFilesStorage or, if you have a custom storage, inherit from CachedFilesMixin. Also, it is best to configure your caches to use the local memory cache backend to perform the static filename to its hashed name lookup. Use a static asset manager: An asset manager can pre-process your static assets to minify, compress, or concatenate them, thereby reducing their size and minimizing requests. It can also preprocess them, enabling you to write them in other languages, such as CoffeeScript and Syntactically awesome stylesheets (Sass). There are several Django packages that offer static asset management such as django-pipeline or webassets.

[ 730 ]

Production-Ready

Chapter 27

Backend performance The scope of backend performance improvements covers your entire server-side web stack, including database queries, template rendering, caching, and background jobs. You will want to extract the highest performance from them since it is entirely within your control. For quick and easy profiling needs, django-debug-toolbar is quite handy. We can also use Python profiling tools, such as the hotshot module for detailed analysis. In Django, you can use one of the several profiling middleware snippets to display the output of hotshot in the browser. A recent live-profiling solution is django-silk. It stores all the requests and responses in the configured database, allowing aggregated analysis over an entire user session, say, to find the worst-performing views. It can also profile any piece of Python code by adding a decorator. As before, we will take a look at some of the ways to improve backend performance. However, considering that they are vast topics in themselves, they have been grouped into sections. Many of these have already been covered in the previous chapters but have been summarized here for easy reference.

Templates As the documentation suggests, you should enable the cached template loader in production. This avoids the overhead of reparsing and recompiling the templates each time it needs to be rendered. The cached template is compiled the first time it is needed and then stored in memory. Subsequent requests for the same template are served from memory. If you find that another templating language such as Jinja2 renders your page significantly faster, then it is quite easy to replace the built-in Django template language.

[ 731 ]

Production-Ready

Chapter 27

Database Sometimes, the Django ORM can generate inefficient SQL code. There are several optimization patterns to improve this, as follows: Reduce database hits with select_related: If you are using a OneToOneField or a Foreign key relationship, in forwarding direction, for a large number of objects, then select_related() can perform a SQL join and reduce the number of database hits. Reduce database hits with prefetch_related: For accessing a ManyToManyField method or, a Foreign key relation, in reverse direction, or a Foreign key relation in a large number of objects, consider using prefetch_related to reduce the number of database hits. Fetch only needed fields with values or values_list: You can save time and memory usage by limiting queries to return only the needed fields and skipping model instantiation using values() or values_list(). Denormalize models: Selective denormalization improves performance by reducing joins at the cost of data consistency. It can also be used for precomputing values, such as the sum of fields or the active status report into an extra column. Compared to using annotated values in queries, denormalized fields are often simpler and faster. Add an index: If a non-primary key gets searched a lot in your queries, consider setting that field's db_index to True in your model definition. Create, update, and delete multiple rows at once: Multiple objects can be operated upon in a single database query with the bulk_create(), update(), and delete() methods. However, they come with several important caveats such as skipping the save() method on that model. So, read the documentation carefully before using them. As a last resort, you can always fine-tune the raw SQL statements using proven database performance expertise. However, maintaining the SQL code can be painful over time.

[ 732 ]

Production-Ready

Chapter 27

Caching Any computation that takes the time can take advantage of caching and return precomputed results faster. However, the problem is stale data or, often, quoted as one of the hardest things in computer science, cache invalidation. This is commonly spotted when, despite refreshing the page, a YouTube video's view count doesn't change. Django has a flexible cache system that allows you to cache anything from a template fragment to an entire site. It allows a variety of pluggable backends such as file-based or data-based backed storage. Most production systems use a memory-based caching system, such as Redis or Memcached. This is purely because volatile memory is many orders of magnitude faster than disk-based storage. Such cache stores are ideal for storing frequently used but ephemeral data, such as user sessions.

Cached session backend By default, Django stores its user session in the database. This usually gets retrieved for every request. To improve performance, the session data can be stored in memory by changing the SESSION_ENGINE setting. For instance, add the following in settings.py to store the session data in your cache: SESSION_ENGINE = "django.contrib.sessions.backends.cache"

Since some cache storage can evict stale data leading to the loss of session data, it is preferable to use Redis or Memcached as the session store, with memory limits high enough to support the maximum number of active user sessions.

Caching frameworks For basic caching strategies, it might be easier to use a caching framework. Among the popular ones are django-cache-machine and django-cachalot. They can handle common scenarios, such as automatically caching results of queries to avoid database hits every time you perform a read.

[ 733 ]

Production-Ready

Chapter 27

The simplest of these is Django-cachalot, a successor of Johnny Cache. It requires very little configuration. It is ideal for sites that have multiple reads and infrequent writes (that is, the vast majority of applications), it caches all Django ORM-read queries in a consistent manner.

Caching patterns Once your site starts getting heavy traffic, you will need to start exploring several caching strategies throughout your stack. Using Varnish, a caching server that sits between your users and Django, many of your requests might not even hit the Django server. Varnish can make pages load extremely fast (sometimes, hundreds of times faster than normal). However, if used improperly, it might serve static pages to your users. Varnish can be easily configured to recognize dynamic pages or dynamic parts of a page such as a shopping cart. Russian doll caching, popular in the rails community, is an interesting template cache-invalidation pattern. Imagine a user's timeline page with a series of posts, each containing a nested list of comments. In fact, the entire page can be considered as several nested lists of content. At each level, the rendered template fragment gets cached. So, if a new comment gets added to a post, only the associated post and timeline caches get invalidated. We first invalidate the cache content directly outside the changed content and move progressively until we reach the outermost content. The dependencies between models need to be tracked for this pattern to work. Another common caching pattern is to cache forever. Even after the content changes, the user might get served stale data from the cache. However, an asynchronous job, such as a Celery job, also gets triggered to update the cache. You can also periodically warm the cache at a certain interval to refresh the content. Essentially, a successful caching strategy identifies the static and dynamic parts of a site. For many sites, the dynamic parts are the user-specific data when you are logged in. If this is separated from the generally available public content, then implementing caching becomes easier.

[ 734 ]

Production-Ready

Chapter 27

Don't treat caching as integral to the working of your site. The site must fall back to a slower but working state even if the caching system breaks down. Cranos It was six in the morning and the SHIM building was surrounded by a grey fog. Somewhere inside, a small conference room had been designated the war room. For the last three hours, the SuperBook team had been holed up here diligently executing their pre-go-live plan. More than 30 users had logged on the IRC chatroom #superbookgolive from various parts of the world. The chat log was projected on a giant whiteboard. When the last item was struck off, Evan glanced at Steve. Then, he pressed a key triggering the deployment process. The room fell silent as the script output kept scrolling off the wall. One error, Steve thought, just one error can potentially set them back by hours. Several seconds later, the command prompt reappeared. It was live! The team erupted in joy. Leaping from their chairs they gave high-fives to each other. Some were crying tears of happiness. After weeks of uncertainty and hard work, it all seemed surreal. However, the celebrations were short-lived. A loud explosion from above shook the entire building. Steve knew the second breach had begun. He shouted to Evan, "don't turn on the beacon until you get my message", and sprinted out of the room. As Steve hurried up the stairway to the rooftop, he heard the sound of footsteps above him. It was Madam O. She opened the door and flung herself in. He could hear her screaming "no!" and a deafening blast shortly after that.

[ 735 ]

Production-Ready

Chapter 27

By the time he reached the rooftop, he saw Madam O sitting with her back against the wall. She was clutching her left arm and wincing in pain. Steve slowly peered around the wall. At a distance, a tall bald man seemed to be working on something with the help of two robots. "He looks like...." Steve broke off, unsure of himself. "Yes, it is Hart. Rather I should say he is Cranos now." "What?" "Yes, a split personality. A monster that laid hidden in Hart's mind for years. I tried to help him control it. Many years back, I thought I had stopped it from ever coming back. However, all this stress took a toll on him. Poor thing, if only I could get near him." Poor thing indeed, he nearly tried to kill her. Steve took out his mobile and sent out a message to turn on the beacon. He had to improvise. With his hands high in the air and fingers crossed, he stepped out. The two robots immediately aimed directly at him. Cranos motioned them to stop. "Well, who do we have here? Mr. SuperBook himself. Did I crash into your launch party, Steve?" "It was our launch, Hart." "Don't call me that", growled Cranos. "That guy was a fool. He wrote the Sentinel code but he never understood its potential. I mean, just look at what Sentinels can do, unravel every cryptographic algorithm known to man. What happens when it enters an intergalactic network?" The hint was not lost on Steve. "SuperBook?" he asked slowly.

[ 736 ]

Production-Ready

Chapter 27

Cranos let out a malicious grin. Behind him, the robots were busy wiring into SHIM's core network. "While your SuperBook users will be busy playing SuperVille, the tentacles of Sentinel will spread into new unsuspecting worlds. Critical systems of every intelligent species will be sabotaged. The Supers will have to bow to a new intergalactic supervillain Cranos." As Cranos was delivering this extended monologue, Steve noticed a movement of the corner of his eye. It was Acorn, the superintelligent squirrel, scurrying along the right edge of the rooftop. He also spotted Hexa hovering strategically on the other side. He nodded at them. Hexa levitated a garbage bin and flung it towards the robots. Acorn distracted them with high-pitched whistles. "Kill them all!" Cranos said irritably. As he turned to watch his intruders, Steve fished out his phone, dialed into FaceTime and held it towards Cranos. "Say hello to your old friend, Cranos," said Steve. Cranos turned to face the phone and the screen revealed Madam O's face. With a smile, she muttered under her breath, "Taradiddle Bumfuzzle!" The expression on Cranos's face changed instantly. The seething anger disappeared. He now looked like a man they had once known. "What happened?" asked Hart confused. "We thought we had lost you," said Madam O over the phone. "I had to use hypnotic trigger words to bring you back." Hart took a moment to survey the scene around him. Then, he slowly smiled and nodded at her. ---------------------------------------------------One Year Later Who would have guessed Acorn would turn into an intergalactic singing sensation in less than a year? His latest album Acorn Unplugged debuted at the top of Billboard's Top 20 chart. He threw a grand party in his new white mansion overlooking a lake.

[ 737 ]

Production-Ready

Chapter 27

The guest list included superheroes, pop stars, actors, and celebrities of all sorts. "So, there was a singer in you after all," said Captain Obvious holding a martini. "I guess there was," replied Acorn. He looked dazzling in a golden tuxedo with all sorts of bling-bling. Steve appeared with Hexa in tow, who looked ravishing in a flowing silver gown. "Hey Steve, Hexa. It has been a while. Is SuperBook still keeping you late at work, Steve?" "Not so much these days. Knock on wood," replied Hexa with a smile. "Ah, you guys did a fantastic job. I owe a lot to SuperBook. My first single, 'Warning: Contains Nuts', was a huge hit in the Tucana galaxy. They watched the video on SuperBook more than a billion times!" "I am sure every other superhero has a good thing to say about SuperBook too. Take Blitz. His AskMeAnything interview won back the hearts of his fans. They were thinking that he was on experimental drugs all this time. It was only when he revealed that his father was Hurricane that his powers made sense." "By the way, how is Hart doing these days?" "Much better," said Steve. "He got professional help. The sentinels were handed back to S.H.I.M. They are developing a new quantum cryptographic algorithm that will be much more secure." "So, I guess we are safe until the next supervillain shows up," said Captain Obvious hesitantly. "Hey, at least the beacon works," said Steve, and the crowd burst into laughter.

[ 738 ]

Production-Ready

Chapter 27

Summary In this final chapter, we looked at various approaches to make your Django application stable, reliable, and fast. In other words, to make it production-ready. Although system administration might be an entire discipline in itself, a fair knowledge of the web stack is essential. We explored several hosting options, including PaaS, VPS, and Serverless. We also looked at several automated deployment tools and a typical deployment scenario. Finally, we covered several techniques to improve frontend and backend performance. The most important milestone of a website is finishing and taking it to production. However, it is by no means the end of your development journey. There will be new features, alterations, and rewrites. Every time you revisit the code, use the opportunity to take a step back and find a cleaner design, identify a hidden pattern, or think of a better implementation. Other developers, and perhaps your future self, will thank you for it.

[ 739 ]

Other Books You May Enjoy If you enjoyed this book, you may be interested in these other books by Packt:

Mastering Python Networking - Second Edition Eric Chou ISBN: 978-1-78913-5992 Use Python libraries to interact with your network Integrate Ansible 2.5 using Python to control Cisco, Juniper, and Arista eAPI network devices Leverage existing frameworks to construct high-level APIs Learn how to build virtual networks in the AWS Cloud Understand how Jenkins can be used to automatically deploy changes in your network Use PyTest and Unittest for Test-Driven Network Development

Other Books You May Enjoy

Python: End-to-end Data Analysis Phuong Vothihong et al. ISBN: 978-1-78839-469-7 Understand the importance of data analysis and master its processing steps Get comfortable using Python and its associated data analysis libraries such as Pandas, NumPy, and SciPy Clean and transform your data and apply advanced statistical analysis to create attractive visualizations Analyze images and time series data Mine text and analyze social networks Perform web scraping and work with different databases, Hadoop, and Spark Use statistical models to discover patterns in data Detect similarities and differences in data with clustering Work with Jupyter Notebook to produce publication-ready figures to be included in reports

[ 741 ]

Other Books You May Enjoy

Leave a review - let other readers know what you think Please share your thoughts on this book with others by leaving a review on the site that you bought it from. If you purchased the book from Amazon, please leave us an honest review on this book's Amazon page. This is vital so that other potential readers can see and use your unbiased opinion to make purchasing decisions, we can understand what our customers think about our products, and our authors can see your feedback on the title that they have worked with Packt to create. It will only take a few minutes of your time, but is valuable to other potential customers, our authors, and Packt. Thank you!

[ 742 ]

Index A acceptance tests 263 accepted content types 443 active link, template patterns issue details 631 solution details 631 admin interface customizations about 642 base and stylesheets, changing 642 Bootstrap-themed admin 644 complete overhauls 644 heading, changed 642 admin interface using 634, 635 admin feature flags 646 models, enhancing 637, 638, 640 protecting 645 anonymous functions 139 API pattern about 714 human browsable interface 714 infinite scrolling 716 API browsing, with relationships 482, 485 browsing, with resources 482, 485 application testing 262 assertions 269, 363 Asynchronous JavaScript And XML (AJAX) 717 asynchronous patterns about 686 endpoint callback pattern 686 asynchronous scraper 700 Asynchronous Server Gateway Interface

(ASGI) 702 asynchronous solutions Celery 688 Django channels 688 for Django 688 asynchronous web-scraping 698, 699 asynchronous, pitfalls deadlock 686 debugging challenge 686 order preservation 686 race condition 686 starvation 686 asynchronous code, pitfalls 685 need for 685 asyncio module reference 348 asyncio about 695 classic web-scraper, example 697 concurrency 701 parallelism 701 versus threads 696 attribute shadowing 195, 196 authenticated HTTP PATCH requests creating, with Postman 549 authenticated requests creating 545, 548 authentication classes 530

B backend performance about 731 caching 733 database 732 templates 731

base and stylesheets changing 642 rich-text editor, adding for WYSIWYG editing 643 base image 722 batch sudoku-solver about 334 implementing, in Python 335, 336, 337, 338, 339 solving, with multiprocessing 340, 341, 342, 343 Sudoku 334, 335 binary mode files, reading 229 files, writing 229 binary search 365 Bitbucket reference 291 black-box tests 262 Booleans 51 Bootstrap about 626 lightweight alternatives 627 manually copy 626 package, using 626 project skeleton, finding 625 URL, for downloading 626 using 625 boundary 280 branching 85 break statement 99 browsable API HTTP DELETE requests, creating 476 HTTP GET requests, creating 465 HTTP OPTIONS requests, creating 474 HTTP POST requests, creating 468 HTTP PUT requests, creating 471 used, for pagination testing 522 used, for testing filter functionality 522 used, for testing order functionality 522 used, for testing search functionality 522 built-in data types collections module 72 immutable sequences 56 mapping types 68, 69 mutable sequences 61

numbers 49 set types 66 built-in exceptions hierarchy reference 291 built-in functions 141 built-in types generation behavior 180 byte arrays 65 bytes 56

C caching, backend performance cached session backend 733 caching frameworks 733 caching patterns 734 case examples, concurrent execution batch sudoku-solver 334 concurrent mergesort 328 random pictures, downloading 343, 344, 345 Celery best practices 691 failure, handling 691 idempotent tasks 692 passing complex objects, avoid to tasks 695 working 689 working with 688 writing, avoid to shared or global state 693 ChainMap 75 classes 40 Classes section, Python tutorial reference 39 classic web-scraper asynchronous web-scraping 698, 699 example 697 synchronous web-scraping 697 Clickjacking about 676 Django, need for 677 closures 354 code documenting 143 guidelines 42 coding 10, 11 collections module about 72

[ 744 ]

ChainMap 75 defaultdict 74 namedtuple 73 combinatoric generators 111 complex numbers 54 composition 199, 201, 203 comprehensions about 150, 155, 156 avoiding 175, 176, 178 filtering 157, 158 computer programming 10 concurrency 701 about 299 versus parallelism 299 concurrent execution case examples 328 events, sending 320, 321 in Python 309 inter-process communication, with queues 321, 322 local data, implementing for thread 317, 318 multiple threads, spawning 314, 315 process communication 318 process pools 322, 324, 325 process, starting 312 process, stopping 312, 313 process, using for timeout addition to function 325, 327 race conditions, dealing with 315, 316, 317 thread communication 318, 319, 320 thread pools 322, 324, 325 thread, starting 309, 310, 311 thread, stopping 312, 313 concurrent mergesort about 328 multiprocess mergesort 332, 333 multithreaded mergesort 331, 332 single-thread mergesort 329, 330 single-thread multipart mergesort 330, 331 conditional programming about 85, 86 elif 86, 87 ternary operator 88 console 26 console editors

using 364 constraint propagation 335 constructor 197 content delivery network (CDN) 730 content negotiation classes advantages 451 content types working with 455 context manager used, for opening files 228 context-switching 302 contextlib URL 228 continue statement 100 controlling 401 CPython reference link 173 Create, Read, Update and Delete (CRUD) 390, 415 Cross-Site Request Forgery (CSRF) 415 cross-site request forgery (CSRF) 658 Cross-site request forgery (CSRF) about 673 Django, avoiding 674 Django, need for 674 Cross-site scripting (XSS) about 669, 670 cookies, valuable 671 Django, avoiding 673 Django, need for 672 CRUD views, form patterns issue details 665 solution details 666, 667, 668 CSV generator testing 270, 271, 272, 274, 275, 276, 277, 279 Curl installing 381 URL 381 custom exceptions 291 custom function debugging 352, 354 custom iterator writing 222, 224 customized pagination classes

[ 745 ]

working with 503 customized permission classes object-level permissions 535 Cygwin URL 381

D data classes, Python reference link 221 Data Compression and Archiving URL 236 data interchange formats about 236 custom decoding, with JSON 240, 241, 243 custom encoding, with JSON 240, 241, 243 JSON, working with 237, 238, 240 Data Model, official Python documentation reference 13 data structures selecting 78, 79 data persisting, on disk 249 saving, to database 253, 254, 257, 259 saving, with shelve 251, 252 serializing, with pickle 249, 250 database management systems (DBMS) 256 database analyzing 398 data, saving 253, 254, 257, 259 Django table, generation 400 updating, without race conditions 694 deadlocks 305, 306 debugging information, obtaining 363 log files, inspecting 360, 361, 362 other techniques 362 techniques 352 traceback, inspecting 354, 355, 357 via profiling 363 with assertions 363 with custom function 352, 354 with print function 352 with Python debugger 357, 358, 359 decimal numbers 55 decorate-sort-undecorate

using, URL 153 decoration 187 decorator factory 190, 191 decorators about 184, 186, 187, 189, 190 reference link 187 used, for enabling parsers 449 used, for enabling renderers 449 working, as wrappers 447 default values 126 defaultdict 74 deployment tools about 726 configuration management 727 fabric 726 deserialization about 401 exploring 404, 408 destructive tests 263 determinist profiling 293 development server HTTP DELETE requests, creating 432 HTTP GET requests, creating to target instances collection 420, 426 HTTP GET requests, creating to target single instance 426 HTTP GET requests, creating with Postman 433 HTTP POST requests, creating 428 HTTP PUT requests, creating 429 launching 419 dict comprehensions 159, 160 dictionaries 68, 69 directories compression 236 content, inspecting 235, 236 existence, checking 230 manipulating 231, 233 working with 226 discounts applying 105, 106 dispatcher 108 Django Channels Celery, differences 705 entering 701, 703

[ 746 ]

notifications, listening with WebSockets 703, 704, 705 Django filters types, working with 512 Django REST framework (DRF) about 391, 529 Django Rest framework (DRF) about 710 IDs, hiding 713 Django REST framework (DRF) installing, in isolated environment 375 Django Rest framework (DRF) Public Posts API, improving 710, 712, 713 Django REST framework (DRF) throttling classes, purpose 564 throttling policies, configuring 568 Django shell working with 403, 408 Django Template Language (DTL) about 616 attributes 617 features 616 filters 618 philosophy 619 variables 617 Django views about 415 creating, with serializer classes 413 URLs, routing 418 Django app, creating 376 authentication 529 configurations 378 files 378 folders 378 forms in 650, 651, 652 installing, in isolated environment 375 permissions 529 superuser, creating 539 user, creating 544 Docker about 721 microservices 722 Dockerfile 722 docstrings 143

don't repeat yourself (DRY) principle 32 double-precision floating-point format reference 52 dunder methods 165 dynamic form generation, form patterns issue details 659 solution details 660

E elif condition 86, 87 else clause 101, 102 endpoint callback pattern about 686 polling pattern 687 publish-subscribe pattern 687 enums 76 exception 101, 287, 289 export function testing 280, 281

F fabric, deployment tools deployment steps 727 feature flags, admin A/B testing 647 issue details 646 limit externalities 648 performance testing 647 solution details 646 trails 647 Fibonacci sequence example 181, 182 files checking 230 compression 236 manipulating 231, 233 opening 226, 227 opening, with context manager 228 overriding, protecting against 230 reading 228 writing 228 filter backend classes configuring 506 filter function 154 filter functionality

[ 747 ]

adding 509 testing, with browsable API 522 filters 360 for loop 89 form patterns about 659 CRUD views 665 dynamic form generation 659 multiple form actions, handling in view 662 user-based forms 661 form processing with class-based views 658 formatted string literals 59 formatters 360 forms crisp 657 data cleaning 654 displaying 656 empty form 649 filled form 649 in Django 650, 651, 652 submitted form without errors 650 submitted form, with errors 650 working 649 fractions 54 front-end tests 262 frontend performance cache infinitely 730 static asset manager 730 Function as a Service (FaaS) 725 functional tests 263 functions about 32, 113, 114 anonymous functions 139 attributes 140 benefits 114 built-in functions 141 code duplication, reducing 115 complex task, splitting 116 example 142 implementation details, hiding 116 readability, improving 117 recursive functions 138 tips 137 traceability, improving 118

URLs, routing 418

G generalizations unpacking 132 generator expressions 169, 170, 172 generator functions 161, 162, 164, 165, 167, 168 generators about 161 avoiding 175, 176, 178 GitHub reference 291 global interpreter lock (GIL) 303, 701 global statement 120 granularity 280 graphical user interface (GUI) 29 Graphical User Interface (GUI) 384 Graphite 729 gray-box testing 262 GUI application Python, running as 29 Gunicorn/Django container 723

H handlers 360 hashability 66 Haskell reference link 155 hosting about 723 approaches 725 Platform as a Service (PaaS) 723 serverless 724 virtual private server (VPS) 724 HTTP DELETE requests creating 432 creating, with browsable API 476 HTTP GET requests creating, to target instance collection 420, 426 creating, to target single instance 426 creating, with browsable API 465 creating, with Postman 433 HTTP OPTIONS requests

[ 748 ]

creating, with browsable API 474 HTTP POST requests creating 428 creating, with browsable API 468 creating, with Postman 435 HTTP PUT requests creating 429 creating, with browsable API 471 HTTP requests creating 246, 247, 248 sending, with unsupported HTTP verbs 457 HTTPie installing 383 human browsable interface, API pattern issue details 715 solution details 715

I iCurlHTTP installing 386 references 386 immutable sequences about 56 bytes 56 strings 56 tuples 59, 61 imports relative imports 146 in-memory stream using 244, 245 indexing 57, 80 infinite iterators 109 infinite loop 98 infinite scrolling, API pattern issue details 717 solution details 717 inheritance 199, 201, 203 initializer 41, 197 input parameters about 122 combining 130, 131 considerations 123, 124 keyword arguments 126 keyword-only arguments 130 positional arguments 125

specifying 125 variable keyword arguments 128 variable positional arguments 127 input/output 244 instance attributes 194 integer division 50 integers 49, 50 Integrated Development Environments (IDEs) 28, 44, 45, 218 integration tests 263 inter-process communication (IPC) 307 iterable 92, 222 iterators about 93, 223 terminating, on shortest input sequence 110 itertools module reference 109

J JavaScript Object Notation (JSON) 393 about 237 custom decoding 240, 241, 243 custom encoding 240, 241, 243 URL 237 Jenkins box 266 Jinja2 about 620 autoescape 620 customizability 620 familiarity 620 performance 620 whitespace control 620

K keyword arguments 126 keyword-only arguments 130

L lambdas 139 library 33 list comprehension 62 lists 61 local, enclosing, global, built-in (LEGB) 37, 119 locks using 305

[ 749 ]

using, with race condition 305 log files about 360 inspecting 360, 361, 362 loggers 360 looping about 89 break statement 99 continue statement 100 else clause 101, 102 for loop 89 while loop 96, 98

M magic methods 41 map function 150, 151, 153 mapping types 68 metaclasses 40, 193 metaprogramming 193 Method Resolution Order (MRO) 209, 211 methods 12 microservice architectures 361 migration about 397 executing 395 mixin classes 495 mocks 269 model serializers advantages 441 models enhancing, for admin 637, 638, 640 modules using 31 monitoring 728 multiple form actions, form pattern issue details 663 solution details 663 multiple sequences iterating over 94, 95 multiple values returning 136 multiprocessing about 300 advantages 308 multithreading

about 300 advantages 308 mutable defaults 133 mutable object 354 mutable sequences about 61 byte arrays 65, 66 lists 61

N name localization 179, 180 name mangling 215 name resolution 120 namedtuple 73 NameError exception 36 names 34, 35, 81 namespace 35 nano using 364 negative indexing 81 nested comprehensions 156 Nginx container 723 nonlocal statements 121, 122 numbers about 49 Booleans 51 complex numbers 54 decimal numbers 55 fractions 54 integers 49, 50 real numbers 52, 53 NumPy about 17

O object-level permissions working with 535 Object-oriented programming (OOP) about 192 attribute, shadowing 195, 196 base class, accessing 203, 205 class methods 211, 213, 215 class namespaces 194 composition 198, 201, 203 data classes 221

[ 750 ]

inheritance 198, 201, 203 instance, initializing 197 Method Resolution Order (MRO) 209, 211 multiple inheritance 206, 207 name mangling 215, 216, 217 object namespaces 194 operator overloading 220 polymorphism 221 private methods 215, 216, 217 property decorator 218, 219 Python class, writing 193 self variable, using 196, 197 static methods 211, 212 Object-Relational Mapping (ORM) 391 object-relational mapping (ORM) 254 objects about 12, 13, 39, 40, 47 immutable 13, 48 importing 144, 145 mutable 13, 48 operator overloading 63, 220 order functionality adding 509 testing, with browsable API 522

P package about 30 using 33 pagination classes configuring 495 pagination about 494 testing, with browsable API 522 Pandas about 17 parallelism 701 about 299 versus concurrency 299 patching 269 pathnames manipulating 233 penetration tests 263 PEP 3134 reference 355

PEP 373 reference 18 PEP 405 virtual environment, creating 368 PEP 448 reference 132 PEP 8 42 reference 144 PEP428 URL 231 performance considerations 172, 173, 175 performance tests 263 performance backend performance 731 frontend performance 730 improving 729 permission policies setting 538 permissions-related data including, to models 531 pickle used, for data serializing 249, 250 Platform as a Service (PaaS) 723 polymorphism 221 positional arguments 125 post/redirect/get (PRG) pattern 658 PostgreSQL container 723 Postman app URL 384 Postman REST client installing 384 Postman authenticated HTTP PATCH requests, creating 549 HTTP POST requests, creating 433, 435 prime generator 103, 104 prime number 103 print function used, for debugging 352 process about 300 anatomy 306 properties 307 production environment about 719

[ 751 ]

stack components 721 web stack, selecting 720 profiling 295, 363 properties 12 property decorator 218, 219 PyPy reference 17 pytest unit testing 590, 591, 592 unit tests, discovering 599, 600, 601, 602 unit tests, executing 599, 600, 601, 602, 608, 609 Python 2 versus Python 3 18, 19 Python 3.x virtual environment, creating 368 Python code organizing 30, 31 Python debugger using 357, 358, 359 Python interactive shell running 27, 28 Python interpreter setting up 20, 21 Python Package Index (PyPI) 15 Python program running 26 Python scripts running 26, 27 Python Tutor reference 47 Python about 14 coherence 15 concurrent execution 309 culture 43 developer productivity 15 drawbacks 16, 17 environment, setting up 18 execution model 34 extensive library 15 installing 19 portability 14 profiling 293, 294, 295 reference 20, 359

running, as GUI application 29 running, as service 28 satisfaction 16 software integration 16 software quality 16 users 17 Pythonic 43

Q quality assurance (QA) 262

R race condition about 304 locks, using 305 scenario 304 Rackspace 725 random pictures, downloading example 343, 344, 345 with asyncio 345, 346, 347, 348 range iterating over 90 real numbers 52, 53 recursive functions 138 regression tests 263 relational algebra 254 relational model 253 relative imports about 146 reference 146 Representational state transfer (REST) 707 request methods 415 requests 244 composing, to filter results 517 composing, to order results 517 creating, for results pagination 498 creating, paginated results used 505 creating, to filter results 516 creating, to perform starts with searches 521 REST architectural error codes 708 hypermedia 708 request operations 708 resources 708 RESTful API

[ 752 ]

about 707 design 709 versioning 709 RESTful system cacheable 708 client-server 708 code on demand 708 layered system 708 stateless 708 uniform interface 708 RESTful Web Service about 529 unit tests, writing 593, 595, 596, 597, 598 return values 134, 135 returned content types 443 rich-text editor adding, for WYSIWYG editing 643

S scenario tests 262 Schwartzian transform 151 scopes about 36, 119 built-in scope 36 enclosing scope 36 example 37, 38, 39 global scope 36 local scope 36 search algorithm 335 search functionality adding 509 secured API browsing, with authentication 552 security checklist 681 security including, to models 531 sequence iterating over 91 serialization about 401 exploring 403, 408 service-oriented architecture (SOA) 360 service Python, running as 28 session hijacking 672

set comprehensions 160 set types 66 shell injection about 677 Django, used 678 web attacks 678 shelve used, for saving data 251, 252 single point of failure (SPOF) 720 slicing 57, 80 small values caching 78 smoke tests 263 solution details, active link custom tags 632 template-only solution 631 solution details, multiple form actions separate views, for separate actions 663 view, for separate actions 663 SQL injection about 674 Django, avoiding 676 Django, need for 675 SQLite URL 398 state attributes, form is_bound 650 statistical profiling 293 sticky mode 357 Stoplight installing 385 URL 385 streams 244 strings about 56 decoding 57 encoding 57 formatting 58, 59 Structured Query Language (SQL) 254 superuser creating, for Django 539 supported HTTP OPTIONS requests creating, with command-line tools 453 synchronous scraper 700 synchronous web-scraping 697 Syntactically awesome stylesheets (Sass) 730

[ 753 ]

system-exiting exceptions 291

T tags 618 template inheritance tree, template patterns issue details 628 solution details 629, 630 template patterns about 628 active link 631 template inheritance tree 628 templates organizing 621 working 622, 623 temporary directories 234 temporary files 234 ternary operator 88 test-driven development (TDD) about 285 benefits 286 Green phase 285 Red phase 285 Red-Green-Refactor 285 shortcomings 287 testing guidelines 265, 266 tests acceptance tests 263 anatomy 264 destructive tests 263 execution 264 fixtures 265 front-end tests 262 functional tests 263 integration tests 263 penetration tests 263 performance tests 263 preparation 264 regression tests 263 scenario tests 262 setup 265 smoke tests 263 teardown 265 unit tests 263 usability tests 263

user acceptance testing (UAT) 263 using, for debug 365 verification 265 text/HTML content rendering possibility 460 thread, states dead 301 new thread 301 not-running 301 runnable 301 running 301 thread about 300 anatomy 300 context-switching 302 deadlocks 303 global interpreter lock (GIL) 302 kernel-level threads 300 killing 301 race condition 303, 304 user-level threads 300 threading module reference 317 throttling policies testing 571, 575 throttling rules advantages 564 thundering herds 691 Timsort 64 Tkinter about 29 token-based authentication working with 555 tokens generating 558 using 558 Tool Command Language (Tcl) 29 tools Curl, installing 380 HTTPie, installing 383 iCurlHTTP, installing 386 installing 380 Postman REST client, installing 384 Stoplight, installing 385 Toy model

[ 754 ]

creating 394 traceback inspecting 354, 355, 357 triangulation 285 troubleshooting console editors, using 364 debugging breakpoints, determining 364 guidelines 364 monitoring 365 tests, using for debug 365 true division 50 tuples 59, 61 types, generators generator expressions 161 generator functions 161

UTF-8 encoding 57

V

U Unicode code points 56 Uniform Resource Identifiers (URIs) 708 unique constraints defining 486, 491 working with 491, 493 unit testing with pytest 590, 591, 592 unit tests about 263, 267 discovering, with pytest 599, 600, 601, 602 executing, with pytest 599, 600, 601, 602, 608, 609 writing 267, 268 writing, for RESTful Web Service 593, 595, 596, 597, 598 writing, to enhance tests code coverage 603, 604, 606, 607 unpacking 128 unsupported HTTP OPTIONS requests creating, with command-line tools 444 upcasting 52 usability tests 263 user acceptance testing (UAT) 263 user information saving 537 user-based forms, form pattern issue details 661 solution details 662

variable keyword arguments 128 variable positional arguments 127 venv module URL 368 versioning classes 576 versioning scheme configuring 578, 582 versioning, RESTful API custom header versioning 709 media type versioning 710 query string versioning 709 URI versioning 709 versioning testing 582 vim using 364 virtual environment (virtualenv) about 22 creating 23, 24, 25 reference 23 virtual environment activating 371, 374 creating, with PEP 405 368 creating, with Python 3.x 368 deactivating 374 directory structure 370 virtual machines 721 virtual private server (VPS) 724 virtualenv URL 368

W web browser used, for working with web service 463 WebHook 686 while loop 96, 98 white-box tests 262

Y yield from expression 168

[ 755 ]

Z

zip function 150, 153