[{"data":1,"prerenderedAt":2420},["ShallowReactive",2],{"all-posts":3},[4,196,1839],{"path":5,"title":6,"created":7,"tags":8,"excerpt":10,"body":20},"\u002Fposts\u002Fgitops-demystified-elevating-data-engineering-with-ease","GitOps Demystified: Elevating Data Engineering with Ease","2023-07-08",[9],"gitops",{"type":11,"value":12},"minimark",[13,17],[14,15,16],"p",{},"In this blog post I'd like to introduce you to the concept of GitOps.",[14,18,19],{},"If you're a data engineer curious about this buzzword, you've come to the right place.\nWe'll discover what GitOps is all about, if it is somehow linked to DevOps, and why it can be a game-changer for data engineering teams.",{"type":11,"value":21,"toc":185},[22,24,26,31,34,38,41,47,50,53,56,61,64,67,73,76,81,84,89,92,97,100,103,119,122,129,133,136,169,173,176,179,182],[14,23,16],{},[14,25,19],{},[27,28,30],"h2",{"id":29},"what-is-gitops","What is GitOps?",[14,32,33],{},"GitOps is like a powerful way of managing infrastructure and application deployments.\nAt its core, GitOps combines the power of Git, the widely popular version control system, with the principles of infrastructure as code.\nThis means that instead of making manual changes to your infrastructure, you describe and manage it through code stored in a Git repository.\nExciting, right?",[27,35,37],{"id":36},"gitops-vs-devops-are-they-related","GitOps vs. DevOps: Are they related?",[14,39,40],{},"You might be wondering:",[42,43,44],"blockquote",{},[14,45,46],{},"Is GitOps related to DevOps?",[14,48,49],{},"Well, DevOps is a cultural and operational philosophy that brings together development and operations teams to collaborate seamlessly throughout the software development lifecycle.\nIt emphasizes continuous integration, continuous delivery, and collaboration.\nOn the other hand, GitOps is an operational framework that specifically focuses on using Git as the source of truth for managing infrastructure and application deployments.",[14,51,52],{},"In other words, while DevOps is a broader philosophy, GitOps is a specific approach within the DevOps realm that emphasizes using Git as the central control plane for managing deployments.\nWith GitOps, you can ensure that your infrastructure and application configurations are versioned, auditable, and easily reproducible, all through the power of Git.",[14,54,55],{},"At this point you might be a little confused, thinking:",[42,57,58],{},[14,59,60],{},"Isn't versioning my Terraform\u002FAnsible code on Git already applying GitOps?",[14,62,63],{},"This is a great question that highlights an important distinction between Version Control and actual GitOps.",[14,65,66],{},"While versioning your infrastructure code with tools like Terraform or Ansible is indeed a step in the right direction, GitOps takes it a step further by introducing a declarative and continuous approach to infrastructure management:",[14,68,69],{},[70,71,72],"strong",{},"1- GitOps emphasizes the declarative nature of managing infrastructure:",[14,74,75],{},"In GitOps, you simply define the desired state of your infrastructure and application deployments in code stored in a Git repository.\nThis declarative approach ensures that your infrastructure is always aligned with the defined state, regardless of the underlying tools used for provisioning.",[14,77,78],{},[70,79,80],{},"2- GitOps embraces the principles of continuous integration and continuous delivery:",[14,82,83],{},"It leverages Git's capabilities to automate the deployment of infrastructure and application changes. With GitOps, every change committed to the Git repository triggers an automated workflow to deploy those changes, ensuring a consistent and reproducible deployment process.",[14,85,86],{},[70,87,88],{},"3- GitOps goes beyond version control by treating Git as the single source of truth for your infrastructure and application deployments:",[14,90,91],{},"It promotes collaboration, visibility, and accountability among team members. Everyone can access, review, and contribute to the codebase, enabling efficient collaboration and reducing manual errors.",[93,94,96],"h3",{"id":95},"let-me-stress-this","Let me stress this:",[14,98,99],{},"The declarative nature of GitOps allows you to only worry about defining the desired state of your infrastructure, not about how to reach it.\nThere will be a service that either will be noticed by using git hooks, or that will keep polling the Git repository for new changes, and when such changes are found, an automatic process to reflect those changes in your infrastructure will start applying them.",[14,101,102],{},"This approach is more secure and less prone to human error:",[104,105,106,113],"ul",{},[107,108,109,112],"li",{},[70,110,111],{},"More secure"," because the GitOps service reacting to changes in your repository will only need read-only permissions to read it.",[107,114,115,118],{},[70,116,117],{},"Less prone to human error"," because you don't need to implement any custom imperative CI\u002FCD pipelines, the GitOps service will take care of that.",[14,120,121],{},"The below image clearly shows how a GitOps flow would work:",[14,123,124],{},[125,126],"img",{"alt":127,"src":128},"GitOps flow","\u002Fimages\u002Fgitops-articles\u002Fgitops-flow.png",[27,130,132],{"id":131},"how-can-gitops-help-data-engineering-teams","How can GitOps help Data Engineering teams?",[14,134,135],{},"Now, let's get to the exciting part: how GitOps can benefit data engineering teams.",[137,138,139,145,151,157,163],"ol",{},[107,140,141,144],{},[70,142,143],{},"Leverage of Version Control:"," With GitOps, you will drastically reduce those moments when you accidentally make changes to your infrastructure that you wish you could undo. By treating your infrastructure as code and leveraging Git's version control, you gain the ability to track changes, roll back to previous states, and collaborate efficiently with your team.",[107,146,147,150],{},[70,148,149],{},"Reproducibility and Consistency:"," Data engineering often involves complex pipelines with multiple dependencies. GitOps allows you to define your data pipeline configurations as code, ensuring reproducibility across different stages. You can easily create consistent environments, ensuring that everyone on your team works with the same setup, minimizing surprises and headaches.",[107,152,153,156],{},[70,154,155],{},"Streamlined Collaboration:"," GitOps fosters collaboration within your data engineering team. By storing your infrastructure and application code in a Git repository, everyone can easily access, review, and contribute to the codebase. The days of tangled email threads and confusion are over. GitOps promotes transparency and enables smooth teamwork.",[107,158,159,162],{},[70,160,161],{},"Effortless Auditing and Compliance:"," Security and compliance are crucial in the realm of data engineering. GitOps makes it easier to meet these requirements. Since all changes are versioned and traceable, you can easily audit and analyze every modification made to your infrastructure code. This not only enhances compliance efforts but also strengthens your overall data governance practices.",[107,164,165,168],{},[70,166,167],{},"Disaster Recovery and Rollback:"," Data engineering processes can be complex, and unexpected failures can have a severe impact on data integrity. GitOps allows data engineering teams to roll back to a known working state effortlessly. This capability ensures quick recovery from failures and reduces the risk of data loss or corruption.",[27,170,172],{"id":171},"conclusion","Conclusion",[14,174,175],{},"We have learnt how GitOps is a powerful approach for managing infrastructure and application deployments, leveraging Git as the source of truth.",[14,177,178],{},"And as a data engineer, you have discovered how GitOps can streamline your workflows, enhance collaboration, and bring greater security and compliance to your data pipelines.",[14,180,181],{},"In the upcoming articles I will showcase and dive more deeply into some tools that will enable you to manage your infrastructure, your services and even ETL pipelines using GitOps principles.",[14,183,184],{},"Stay tuned for more GitOps.",{"title":186,"searchDepth":187,"depth":187,"links":188},"",2,[189,190,194,195],{"id":29,"depth":187,"text":30},{"id":36,"depth":187,"text":37,"children":191},[192],{"id":95,"depth":193,"text":96},3,{"id":131,"depth":187,"text":132},{"id":171,"depth":187,"text":172},{"path":197,"title":198,"created":199,"tags":200,"excerpt":202,"body":207},"\u002Fposts\u002Fchanging-code-almost-without-editing-it-the-magic-of-decorators","Changing code - almost - without editing it: the magic of decorators","2019-09-29",[201],"python",{"type":11,"value":203},[204],[14,205,206],{},"As a Data Engineer, one of the common tasks is to support the Data Science team.\nSometimes you are required to customize one of the products they use daily.\nHow to make these customizations the least invasive, so that you can easily update it?",{"type":11,"value":208,"toc":1830},[209,211,215,226,229,232,235,239,253,262,979,990,1001,1005,1008,1014,1312,1315,1318,1323,1326,1329,1337,1340,1343,1347,1355,1360,1365,1368,1491,1494,1502,1505,1509,1512,1519,1710,1717,1724,1743,1749,1772,1775,1778,1794,1798,1801,1805,1826],[14,210,206],{},[27,212,214],{"id":213},"problem","Problem",[14,216,217,218,225],{},"Let's say your data science team is using ",[219,220,224],"a",{"href":221,"rel":222},"http:\u002F\u002Fgethue.com\u002F",[223],"nofollow","Hue"," to perform data exploration activities on the datasets in your data lake.\nThey find the dataset they want, they understand how to extract the information they want from it, so they create a wonderful query and want to save it to a file, for example to schedule it.",[14,227,228],{},"Here comes the pain:",[14,230,231],{},"Queries are saved in the internal Hue database, there is no option to store queries as files somewhere else (e.g. S3 or the local filesystem).",[14,233,234],{},"It is true that one could simply copy-paste the query in a local sql file and version it using git or any other SVC, but that's not the best user experience, especially for someone that is not super technical.",[27,236,238],{"id":237},"exploring-the-hue-codebase","Exploring the Hue codebase",[14,240,241,242,246,247,252],{},"You put aside your ",[243,244,245],"em",{},"data engineer-y"," skills and start digging into the Hue codebase, trying to find the code that saves queries in the hue database. Eventually you find that it is in the ",[219,248,251],{"href":249,"rel":250},"https:\u002F\u002Fgithub.com\u002Fcloudera\u002Fhue\u002Ftree\u002Fmaster\u002Fdesktop\u002Flibs\u002Fnotebook\u002Fsrc\u002Fnotebook",[223],"notebook"," library.",[14,254,255,256,261],{},"After some further digging you discover how Hue ",[219,257,260],{"href":258,"rel":259},"https:\u002F\u002Fgithub.com\u002Fcloudera\u002Fhue\u002Fblob\u002Fbranch-4.4\u002Fdesktop\u002Flibs\u002Fnotebook\u002Fsrc\u002Fnotebook\u002Fapi.py#L361-L376",[223],"saves a notebook",":",[263,264,267],"pre",{"className":265,"code":266,"language":201,"meta":186,"style":186},"language-python shiki shiki-themes github-light github-dark","def _save_notebook(notebook, user):\n  notebook_type = notebook.get('type', 'notebook')\n  save_as = False\n\n  if notebook.get('parentSavedQueryUuid'): # We save into the original saved query, not into the query history\n    notebook_doc = Document2.objects.get_by_uuid(user=user, uuid=notebook['parentSavedQueryUuid'])\n  elif notebook.get('id'):\n    notebook_doc = Document2.objects.get(id=notebook['id'])\n  else:\n    notebook_doc = Document2.objects.create(name=notebook['name'], uuid=notebook['uuid'], type=notebook_type, owner=user)\n    Document.objects.link(notebook_doc, owner=notebook_doc.owner, name=notebook_doc.name, description=notebook_doc.description, extra=notebook_type)\n    save_as = True\n\n    if notebook.get('directoryUuid'):\n      notebook_doc.parent_directory = Document2.objects.get_by_uuid(user=user, uuid=notebook.get('directoryUuid'), perm_type='write')\n    else:\n      notebook_doc.parent_directory = Document2.objects.get_home_directory(user)\n\n  notebook['isSaved'] = True\n  notebook['isHistory'] = False\n  notebook['id'] = notebook_doc.id\n  _clear_sessions(notebook)\n  notebook_doc1 = notebook_doc._get_doc1(doc2_type=notebook_type)\n  notebook_doc.update_data(notebook)\n  notebook_doc.search = _get_statement(notebook)\n  notebook_doc.name = notebook_doc1.name = notebook['name']\n  notebook_doc.description = notebook_doc1.description = notebook['description']\n  notebook_doc.save()\n  notebook_doc1.save()\n\n  return notebook_doc, save_as\n\n\n@api_error_handler\n@require_POST\n@check_document_modify_permission()\ndef save_notebook(request):\n  response = {'status': -1}\n\n  notebook = json.loads(request.POST.get('notebook', '{}'))\n\n  notebook_doc, save_as = _save_notebook(notebook, request.user)\n\n  response['status'] = 0\n  response['save_as'] = save_as\n  response.update(notebook_doc.to_dict())\n  response['message'] = request.POST.get('editorMode') == 'true' and _('Query saved successfully') or _('Notebook saved successfully')\n\n  return JsonResponse(response)\n",[268,269,270,287,311,322,329,347,380,394,415,424,474,510,521,526,539,577,585,595,600,616,630,644,650,668,674,685,706,726,732,738,743,752,757,762,768,774,783,794,820,825,857,862,873,878,893,908,914,966,971],"code",{"__ignoreMap":186},[271,272,275,279,283],"span",{"class":273,"line":274},"line",1,[271,276,278],{"class":277},"szBVR","def",[271,280,282],{"class":281},"sScJk"," _save_notebook",[271,284,286],{"class":285},"sVt8B","(notebook, user):\n",[271,288,289,292,295,298,302,305,308],{"class":273,"line":187},[271,290,291],{"class":285},"  notebook_type ",[271,293,294],{"class":277},"=",[271,296,297],{"class":285}," notebook.get(",[271,299,301],{"class":300},"sZZnC","'type'",[271,303,304],{"class":285},", ",[271,306,307],{"class":300},"'notebook'",[271,309,310],{"class":285},")\n",[271,312,313,316,318],{"class":273,"line":193},[271,314,315],{"class":285},"  save_as ",[271,317,294],{"class":277},[271,319,321],{"class":320},"sj4cs"," False\n",[271,323,325],{"class":273,"line":324},4,[271,326,328],{"emptyLinePlaceholder":327},true,"\n",[271,330,332,335,337,340,343],{"class":273,"line":331},5,[271,333,334],{"class":277},"  if",[271,336,297],{"class":285},[271,338,339],{"class":300},"'parentSavedQueryUuid'",[271,341,342],{"class":285},"): ",[271,344,346],{"class":345},"sJ8bj","# We save into the original saved query, not into the query history\n",[271,348,350,353,355,358,362,364,367,370,372,375,377],{"class":273,"line":349},6,[271,351,352],{"class":285},"    notebook_doc ",[271,354,294],{"class":277},[271,356,357],{"class":285}," Document2.objects.get_by_uuid(",[271,359,361],{"class":360},"s4XuR","user",[271,363,294],{"class":277},[271,365,366],{"class":285},"user, ",[271,368,369],{"class":360},"uuid",[271,371,294],{"class":277},[271,373,374],{"class":285},"notebook[",[271,376,339],{"class":300},[271,378,379],{"class":285},"])\n",[271,381,383,386,388,391],{"class":273,"line":382},7,[271,384,385],{"class":277},"  elif",[271,387,297],{"class":285},[271,389,390],{"class":300},"'id'",[271,392,393],{"class":285},"):\n",[271,395,397,399,401,404,407,409,411,413],{"class":273,"line":396},8,[271,398,352],{"class":285},[271,400,294],{"class":277},[271,402,403],{"class":285}," Document2.objects.get(",[271,405,406],{"class":360},"id",[271,408,294],{"class":277},[271,410,374],{"class":285},[271,412,390],{"class":300},[271,414,379],{"class":285},[271,416,418,421],{"class":273,"line":417},9,[271,419,420],{"class":277},"  else",[271,422,423],{"class":285},":\n",[271,425,427,429,431,434,437,439,441,444,447,449,451,453,456,458,461,463,466,469,471],{"class":273,"line":426},10,[271,428,352],{"class":285},[271,430,294],{"class":277},[271,432,433],{"class":285}," Document2.objects.create(",[271,435,436],{"class":360},"name",[271,438,294],{"class":277},[271,440,374],{"class":285},[271,442,443],{"class":300},"'name'",[271,445,446],{"class":285},"], ",[271,448,369],{"class":360},[271,450,294],{"class":277},[271,452,374],{"class":285},[271,454,455],{"class":300},"'uuid'",[271,457,446],{"class":285},[271,459,460],{"class":360},"type",[271,462,294],{"class":277},[271,464,465],{"class":285},"notebook_type, ",[271,467,468],{"class":360},"owner",[271,470,294],{"class":277},[271,472,473],{"class":285},"user)\n",[271,475,477,480,482,484,487,489,491,494,497,499,502,505,507],{"class":273,"line":476},11,[271,478,479],{"class":285},"    Document.objects.link(notebook_doc, ",[271,481,468],{"class":360},[271,483,294],{"class":277},[271,485,486],{"class":285},"notebook_doc.owner, ",[271,488,436],{"class":360},[271,490,294],{"class":277},[271,492,493],{"class":285},"notebook_doc.name, ",[271,495,496],{"class":360},"description",[271,498,294],{"class":277},[271,500,501],{"class":285},"notebook_doc.description, ",[271,503,504],{"class":360},"extra",[271,506,294],{"class":277},[271,508,509],{"class":285},"notebook_type)\n",[271,511,513,516,518],{"class":273,"line":512},12,[271,514,515],{"class":285},"    save_as ",[271,517,294],{"class":277},[271,519,520],{"class":320}," True\n",[271,522,524],{"class":273,"line":523},13,[271,525,328],{"emptyLinePlaceholder":327},[271,527,529,532,534,537],{"class":273,"line":528},14,[271,530,531],{"class":277},"    if",[271,533,297],{"class":285},[271,535,536],{"class":300},"'directoryUuid'",[271,538,393],{"class":285},[271,540,542,545,547,549,551,553,555,557,559,562,564,567,570,572,575],{"class":273,"line":541},15,[271,543,544],{"class":285},"      notebook_doc.parent_directory ",[271,546,294],{"class":277},[271,548,357],{"class":285},[271,550,361],{"class":360},[271,552,294],{"class":277},[271,554,366],{"class":285},[271,556,369],{"class":360},[271,558,294],{"class":277},[271,560,561],{"class":285},"notebook.get(",[271,563,536],{"class":300},[271,565,566],{"class":285},"), ",[271,568,569],{"class":360},"perm_type",[271,571,294],{"class":277},[271,573,574],{"class":300},"'write'",[271,576,310],{"class":285},[271,578,580,583],{"class":273,"line":579},16,[271,581,582],{"class":277},"    else",[271,584,423],{"class":285},[271,586,588,590,592],{"class":273,"line":587},17,[271,589,544],{"class":285},[271,591,294],{"class":277},[271,593,594],{"class":285}," Document2.objects.get_home_directory(user)\n",[271,596,598],{"class":273,"line":597},18,[271,599,328],{"emptyLinePlaceholder":327},[271,601,603,606,609,612,614],{"class":273,"line":602},19,[271,604,605],{"class":285},"  notebook[",[271,607,608],{"class":300},"'isSaved'",[271,610,611],{"class":285},"] ",[271,613,294],{"class":277},[271,615,520],{"class":320},[271,617,619,621,624,626,628],{"class":273,"line":618},20,[271,620,605],{"class":285},[271,622,623],{"class":300},"'isHistory'",[271,625,611],{"class":285},[271,627,294],{"class":277},[271,629,321],{"class":320},[271,631,633,635,637,639,641],{"class":273,"line":632},21,[271,634,605],{"class":285},[271,636,390],{"class":300},[271,638,611],{"class":285},[271,640,294],{"class":277},[271,642,643],{"class":285}," notebook_doc.id\n",[271,645,647],{"class":273,"line":646},22,[271,648,649],{"class":285},"  _clear_sessions(notebook)\n",[271,651,653,656,658,661,664,666],{"class":273,"line":652},23,[271,654,655],{"class":285},"  notebook_doc1 ",[271,657,294],{"class":277},[271,659,660],{"class":285}," notebook_doc._get_doc1(",[271,662,663],{"class":360},"doc2_type",[271,665,294],{"class":277},[271,667,509],{"class":285},[271,669,671],{"class":273,"line":670},24,[271,672,673],{"class":285},"  notebook_doc.update_data(notebook)\n",[271,675,677,680,682],{"class":273,"line":676},25,[271,678,679],{"class":285},"  notebook_doc.search ",[271,681,294],{"class":277},[271,683,684],{"class":285}," _get_statement(notebook)\n",[271,686,688,691,693,696,698,701,703],{"class":273,"line":687},26,[271,689,690],{"class":285},"  notebook_doc.name ",[271,692,294],{"class":277},[271,694,695],{"class":285}," notebook_doc1.name ",[271,697,294],{"class":277},[271,699,700],{"class":285}," notebook[",[271,702,443],{"class":300},[271,704,705],{"class":285},"]\n",[271,707,709,712,714,717,719,721,724],{"class":273,"line":708},27,[271,710,711],{"class":285},"  notebook_doc.description ",[271,713,294],{"class":277},[271,715,716],{"class":285}," notebook_doc1.description ",[271,718,294],{"class":277},[271,720,700],{"class":285},[271,722,723],{"class":300},"'description'",[271,725,705],{"class":285},[271,727,729],{"class":273,"line":728},28,[271,730,731],{"class":285},"  notebook_doc.save()\n",[271,733,735],{"class":273,"line":734},29,[271,736,737],{"class":285},"  notebook_doc1.save()\n",[271,739,741],{"class":273,"line":740},30,[271,742,328],{"emptyLinePlaceholder":327},[271,744,746,749],{"class":273,"line":745},31,[271,747,748],{"class":277},"  return",[271,750,751],{"class":285}," notebook_doc, save_as\n",[271,753,755],{"class":273,"line":754},32,[271,756,328],{"emptyLinePlaceholder":327},[271,758,760],{"class":273,"line":759},33,[271,761,328],{"emptyLinePlaceholder":327},[271,763,765],{"class":273,"line":764},34,[271,766,767],{"class":281},"@api_error_handler\n",[271,769,771],{"class":273,"line":770},35,[271,772,773],{"class":281},"@require_POST\n",[271,775,777,780],{"class":273,"line":776},36,[271,778,779],{"class":281},"@check_document_modify_permission",[271,781,782],{"class":285},"()\n",[271,784,786,788,791],{"class":273,"line":785},37,[271,787,278],{"class":277},[271,789,790],{"class":281}," save_notebook",[271,792,793],{"class":285},"(request):\n",[271,795,797,800,802,805,808,811,814,817],{"class":273,"line":796},38,[271,798,799],{"class":285},"  response ",[271,801,294],{"class":277},[271,803,804],{"class":285}," {",[271,806,807],{"class":300},"'status'",[271,809,810],{"class":285},": ",[271,812,813],{"class":277},"-",[271,815,816],{"class":320},"1",[271,818,819],{"class":285},"}\n",[271,821,823],{"class":273,"line":822},39,[271,824,328],{"emptyLinePlaceholder":327},[271,826,828,831,833,836,839,842,844,846,849,852,854],{"class":273,"line":827},40,[271,829,830],{"class":285},"  notebook ",[271,832,294],{"class":277},[271,834,835],{"class":285}," json.loads(request.",[271,837,838],{"class":320},"POST",[271,840,841],{"class":285},".get(",[271,843,307],{"class":300},[271,845,304],{"class":285},[271,847,848],{"class":300},"'",[271,850,851],{"class":320},"{}",[271,853,848],{"class":300},[271,855,856],{"class":285},"))\n",[271,858,860],{"class":273,"line":859},41,[271,861,328],{"emptyLinePlaceholder":327},[271,863,865,868,870],{"class":273,"line":864},42,[271,866,867],{"class":285},"  notebook_doc, save_as ",[271,869,294],{"class":277},[271,871,872],{"class":285}," _save_notebook(notebook, request.user)\n",[271,874,876],{"class":273,"line":875},43,[271,877,328],{"emptyLinePlaceholder":327},[271,879,881,884,886,888,890],{"class":273,"line":880},44,[271,882,883],{"class":285},"  response[",[271,885,807],{"class":300},[271,887,611],{"class":285},[271,889,294],{"class":277},[271,891,892],{"class":320}," 0\n",[271,894,896,898,901,903,905],{"class":273,"line":895},45,[271,897,883],{"class":285},[271,899,900],{"class":300},"'save_as'",[271,902,611],{"class":285},[271,904,294],{"class":277},[271,906,907],{"class":285}," save_as\n",[271,909,911],{"class":273,"line":910},46,[271,912,913],{"class":285},"  response.update(notebook_doc.to_dict())\n",[271,915,917,919,922,924,926,929,931,933,936,939,942,945,948,951,954,956,959,961,964],{"class":273,"line":916},47,[271,918,883],{"class":285},[271,920,921],{"class":300},"'message'",[271,923,611],{"class":285},[271,925,294],{"class":277},[271,927,928],{"class":285}," request.",[271,930,838],{"class":320},[271,932,841],{"class":285},[271,934,935],{"class":300},"'editorMode'",[271,937,938],{"class":285},") ",[271,940,941],{"class":277},"==",[271,943,944],{"class":300}," 'true'",[271,946,947],{"class":277}," and",[271,949,950],{"class":285}," _(",[271,952,953],{"class":300},"'Query saved successfully'",[271,955,938],{"class":285},[271,957,958],{"class":277},"or",[271,960,950],{"class":285},[271,962,963],{"class":300},"'Notebook saved successfully'",[271,965,310],{"class":285},[271,967,969],{"class":273,"line":968},48,[271,970,328],{"emptyLinePlaceholder":327},[271,972,974,976],{"class":273,"line":973},49,[271,975,748],{"class":277},[271,977,978],{"class":285}," JsonResponse(response)\n",[14,980,981,982,985,986,989],{},"Interesting, you observe what Hue is passing in the ",[268,983,984],{},"request"," and discover that the ",[268,987,988],{},"_save_notebook"," method contains all the data you need to save the query to a file:",[104,991,992,995,998],{},[107,993,994],{},"the name selected when saving the query",[107,996,997],{},"the query",[107,999,1000],{},"the query type",[27,1002,1004],{"id":1003},"time-to-customize-hue","Time to customize Hue",[14,1006,1007],{},"Now that you feel like a hacker because you discovered how Hue works under the hood, you are tempted to change the function code by adding some lines to save your query to a local file.",[14,1009,1010,1011,261],{},"Something like this ",[70,1012,1013],{},"(This code is untested and doesn't validate the notebook object before using its properties, don't use it in production!!!)",[263,1015,1017],{"className":265,"code":1016,"language":201,"meta":186,"style":186},"@api_error_handler\n@require_POST\n@check_document_modify_permission()\ndef save_notebook(request):\n  response = {'status': -1}\n\n  notebook = json.loads(request.POST.get('notebook', '{}'))\n\n  notebook_doc, save_as = _save_notebook(notebook, request.user)\n\n  # Save query to a local file\n  query_name = notebook['name'].replace(' ', '_')\n  query_type = notebook['type'] # e.g. 'hive-query'\n  destination = '\u002Fpath\u002Fto\u002Fqueries\u002F{}\u002F{}\u002F{}.sql'.format(request.user, query_type, query_name)\n  output_file = open(destination, 'w')\n  output_file.write(str(notebook['snippets'][0]['raw_statement']))\n  output_file.close()\n\n\n  response['status'] = 0\n  response['save_as'] = save_as\n  response.update(notebook_doc.to_dict())\n  response['message'] = request.POST.get('editorMode') == 'true' and _('Query saved successfully') or _('Notebook saved successfully')\n\n  return JsonResponse(response)\n",[268,1018,1019,1023,1027,1033,1041,1059,1063,1087,1091,1099,1103,1108,1132,1148,1175,1193,1221,1226,1230,1234,1246,1258,1262,1302,1306],{"__ignoreMap":186},[271,1020,1021],{"class":273,"line":274},[271,1022,767],{"class":281},[271,1024,1025],{"class":273,"line":187},[271,1026,773],{"class":281},[271,1028,1029,1031],{"class":273,"line":193},[271,1030,779],{"class":281},[271,1032,782],{"class":285},[271,1034,1035,1037,1039],{"class":273,"line":324},[271,1036,278],{"class":277},[271,1038,790],{"class":281},[271,1040,793],{"class":285},[271,1042,1043,1045,1047,1049,1051,1053,1055,1057],{"class":273,"line":331},[271,1044,799],{"class":285},[271,1046,294],{"class":277},[271,1048,804],{"class":285},[271,1050,807],{"class":300},[271,1052,810],{"class":285},[271,1054,813],{"class":277},[271,1056,816],{"class":320},[271,1058,819],{"class":285},[271,1060,1061],{"class":273,"line":349},[271,1062,328],{"emptyLinePlaceholder":327},[271,1064,1065,1067,1069,1071,1073,1075,1077,1079,1081,1083,1085],{"class":273,"line":382},[271,1066,830],{"class":285},[271,1068,294],{"class":277},[271,1070,835],{"class":285},[271,1072,838],{"class":320},[271,1074,841],{"class":285},[271,1076,307],{"class":300},[271,1078,304],{"class":285},[271,1080,848],{"class":300},[271,1082,851],{"class":320},[271,1084,848],{"class":300},[271,1086,856],{"class":285},[271,1088,1089],{"class":273,"line":396},[271,1090,328],{"emptyLinePlaceholder":327},[271,1092,1093,1095,1097],{"class":273,"line":417},[271,1094,867],{"class":285},[271,1096,294],{"class":277},[271,1098,872],{"class":285},[271,1100,1101],{"class":273,"line":426},[271,1102,328],{"emptyLinePlaceholder":327},[271,1104,1105],{"class":273,"line":476},[271,1106,1107],{"class":345},"  # Save query to a local file\n",[271,1109,1110,1113,1115,1117,1119,1122,1125,1127,1130],{"class":273,"line":512},[271,1111,1112],{"class":285},"  query_name ",[271,1114,294],{"class":277},[271,1116,700],{"class":285},[271,1118,443],{"class":300},[271,1120,1121],{"class":285},"].replace(",[271,1123,1124],{"class":300},"' '",[271,1126,304],{"class":285},[271,1128,1129],{"class":300},"'_'",[271,1131,310],{"class":285},[271,1133,1134,1137,1139,1141,1143,1145],{"class":273,"line":523},[271,1135,1136],{"class":285},"  query_type ",[271,1138,294],{"class":277},[271,1140,700],{"class":285},[271,1142,301],{"class":300},[271,1144,611],{"class":285},[271,1146,1147],{"class":345},"# e.g. 'hive-query'\n",[271,1149,1150,1153,1155,1158,1160,1163,1165,1167,1169,1172],{"class":273,"line":528},[271,1151,1152],{"class":285},"  destination ",[271,1154,294],{"class":277},[271,1156,1157],{"class":300}," '\u002Fpath\u002Fto\u002Fqueries\u002F",[271,1159,851],{"class":320},[271,1161,1162],{"class":300},"\u002F",[271,1164,851],{"class":320},[271,1166,1162],{"class":300},[271,1168,851],{"class":320},[271,1170,1171],{"class":300},".sql'",[271,1173,1174],{"class":285},".format(request.user, query_type, query_name)\n",[271,1176,1177,1180,1182,1185,1188,1191],{"class":273,"line":541},[271,1178,1179],{"class":285},"  output_file ",[271,1181,294],{"class":277},[271,1183,1184],{"class":320}," open",[271,1186,1187],{"class":285},"(destination, ",[271,1189,1190],{"class":300},"'w'",[271,1192,310],{"class":285},[271,1194,1195,1198,1201,1204,1207,1210,1213,1215,1218],{"class":273,"line":579},[271,1196,1197],{"class":285},"  output_file.write(",[271,1199,1200],{"class":320},"str",[271,1202,1203],{"class":285},"(notebook[",[271,1205,1206],{"class":300},"'snippets'",[271,1208,1209],{"class":285},"][",[271,1211,1212],{"class":320},"0",[271,1214,1209],{"class":285},[271,1216,1217],{"class":300},"'raw_statement'",[271,1219,1220],{"class":285},"]))\n",[271,1222,1223],{"class":273,"line":587},[271,1224,1225],{"class":285},"  output_file.close()\n",[271,1227,1228],{"class":273,"line":597},[271,1229,328],{"emptyLinePlaceholder":327},[271,1231,1232],{"class":273,"line":602},[271,1233,328],{"emptyLinePlaceholder":327},[271,1235,1236,1238,1240,1242,1244],{"class":273,"line":618},[271,1237,883],{"class":285},[271,1239,807],{"class":300},[271,1241,611],{"class":285},[271,1243,294],{"class":277},[271,1245,892],{"class":320},[271,1247,1248,1250,1252,1254,1256],{"class":273,"line":632},[271,1249,883],{"class":285},[271,1251,900],{"class":300},[271,1253,611],{"class":285},[271,1255,294],{"class":277},[271,1257,907],{"class":285},[271,1259,1260],{"class":273,"line":646},[271,1261,913],{"class":285},[271,1263,1264,1266,1268,1270,1272,1274,1276,1278,1280,1282,1284,1286,1288,1290,1292,1294,1296,1298,1300],{"class":273,"line":652},[271,1265,883],{"class":285},[271,1267,921],{"class":300},[271,1269,611],{"class":285},[271,1271,294],{"class":277},[271,1273,928],{"class":285},[271,1275,838],{"class":320},[271,1277,841],{"class":285},[271,1279,935],{"class":300},[271,1281,938],{"class":285},[271,1283,941],{"class":277},[271,1285,944],{"class":300},[271,1287,947],{"class":277},[271,1289,950],{"class":285},[271,1291,953],{"class":300},[271,1293,938],{"class":285},[271,1295,958],{"class":277},[271,1297,950],{"class":285},[271,1299,963],{"class":300},[271,1301,310],{"class":285},[271,1303,1304],{"class":273,"line":670},[271,1305,328],{"emptyLinePlaceholder":327},[271,1307,1308,1310],{"class":273,"line":676},[271,1309,748],{"class":277},[271,1311,978],{"class":285},[14,1313,1314],{},"This code saves your query to a local file, so you can reference it from external tools.",[14,1316,1317],{},"Good job team, let's go grab some coffee.",[14,1319,1320],{},[70,1321,1322],{},"Not that fast!",[14,1324,1325],{},"The Product Owner of your team rushes into the office and tells you that you need to update to the latest Hue version that has just been released.",[14,1327,1328],{},"It's time to:",[104,1330,1331,1334],{},[107,1332,1333],{},"roll the update",[107,1335,1336],{},"re-apply your customization, plus eventually any other customizations that have been done to Hue.",[14,1338,1339],{},"For some reason you cannot make a Pull Request to the Hue project and you have to maintain this customization on your own.",[14,1341,1342],{},"Wouldn't it be great if there was a way to re-apply your customization to any latest version of Hue?",[27,1344,1346],{"id":1345},"enter-python-decorators","Enter Python Decorators",[14,1348,1349,1350],{},"According to ",[219,1351,1354],{"href":1352,"rel":1353},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPython_syntax_and_semantics#Decorators",[223],"Wikipedia",[42,1356,1357],{},[14,1358,1359],{},"A decorator is any callable Python object that is used to modify a function, method or class definition.",[42,1361,1362],{},[14,1363,1364],{},"A decorator is passed the original object being defined and returns a modified object, which is then bound to the name in the definition.",[14,1366,1367],{},"Decorators are an easy way to extend the behavior of a function by doing something before or after the function is called:",[263,1369,1371],{"className":265,"code":1370,"language":201,"meta":186,"style":186},"def my_decorator(function_to_decorate):\n    def decoration_function(*args, **kwargs):\n        print('Before function call')\n        function_to_decorate(*args, **kwargs)\n        print('After function call')\n    return decoration_function\n\n@my_decorator\ndef decorable_function():\n    print('Inside function call')\n\ndecorable_function()\n",[268,1372,1373,1383,1406,1418,1432,1443,1451,1455,1460,1470,1482,1486],{"__ignoreMap":186},[271,1374,1375,1377,1380],{"class":273,"line":274},[271,1376,278],{"class":277},[271,1378,1379],{"class":281}," my_decorator",[271,1381,1382],{"class":285},"(function_to_decorate):\n",[271,1384,1385,1388,1391,1394,1397,1400,1403],{"class":273,"line":187},[271,1386,1387],{"class":277},"    def",[271,1389,1390],{"class":281}," decoration_function",[271,1392,1393],{"class":285},"(",[271,1395,1396],{"class":277},"*",[271,1398,1399],{"class":285},"args, ",[271,1401,1402],{"class":277},"**",[271,1404,1405],{"class":285},"kwargs):\n",[271,1407,1408,1411,1413,1416],{"class":273,"line":193},[271,1409,1410],{"class":320},"        print",[271,1412,1393],{"class":285},[271,1414,1415],{"class":300},"'Before function call'",[271,1417,310],{"class":285},[271,1419,1420,1423,1425,1427,1429],{"class":273,"line":324},[271,1421,1422],{"class":285},"        function_to_decorate(",[271,1424,1396],{"class":277},[271,1426,1399],{"class":285},[271,1428,1402],{"class":277},[271,1430,1431],{"class":285},"kwargs)\n",[271,1433,1434,1436,1438,1441],{"class":273,"line":331},[271,1435,1410],{"class":320},[271,1437,1393],{"class":285},[271,1439,1440],{"class":300},"'After function call'",[271,1442,310],{"class":285},[271,1444,1445,1448],{"class":273,"line":349},[271,1446,1447],{"class":277},"    return",[271,1449,1450],{"class":285}," decoration_function\n",[271,1452,1453],{"class":273,"line":382},[271,1454,328],{"emptyLinePlaceholder":327},[271,1456,1457],{"class":273,"line":396},[271,1458,1459],{"class":281},"@my_decorator\n",[271,1461,1462,1464,1467],{"class":273,"line":417},[271,1463,278],{"class":277},[271,1465,1466],{"class":281}," decorable_function",[271,1468,1469],{"class":285},"():\n",[271,1471,1472,1475,1477,1480],{"class":273,"line":426},[271,1473,1474],{"class":320},"    print",[271,1476,1393],{"class":285},[271,1478,1479],{"class":300},"'Inside function call'",[271,1481,310],{"class":285},[271,1483,1484],{"class":273,"line":476},[271,1485,328],{"emptyLinePlaceholder":327},[271,1487,1488],{"class":273,"line":512},[271,1489,1490],{"class":285},"decorable_function()\n",[14,1492,1493],{},"Executing the following code will produce the output:",[263,1495,1500],{"className":1496,"code":1498,"language":1499,"meta":186},[1497],"language-text","Before function call\nInside function call\nAfter function call\n","text",[268,1501,1498],{"__ignoreMap":186},[14,1503,1504],{},"With this powerful Python feature, we could save our query locally after it has been stored in the Hue database without raising any Exception.",[27,1506,1508],{"id":1507},"time-to-decorate","Time to decorate!",[14,1510,1511],{},"Knowing all these informations, we can write a decorator that takes our notebook object and saves the query to a local file.",[14,1513,1514,1515,1518],{},"Therefore we create a ",[70,1516,1517],{},"query_decorators.py"," file:",[263,1520,1522],{"className":265,"code":1521,"language":201,"meta":186,"style":186},"def save_query_locally(function_to_decorate):\n    def decoration_function(*args, **kwargs):\n        # Before function call\n        function_to_decorate(*args, **kwargs)\n        # After function call\n        # Save query to a local file\n        notebook = args[0]\n        user = args[1]\n        query_name = notebook['name'].replace(' ', '_')\n        query_type = notebook['type'] # e.g. 'hive-query'\n        destination = '\u002Fpath\u002Fto\u002Fqueries\u002F{}\u002F{}\u002F{}.sql'.format(user, query_type, query_name)\n        output_file = open(destination, 'w')\n        output_file.write(str(notebook['snippets'][0]['raw_statement']))\n        output_file.close()\n    return decoration_function\n",[268,1523,1524,1533,1549,1554,1566,1571,1576,1590,1603,1624,1639,1663,1678,1699,1704],{"__ignoreMap":186},[271,1525,1526,1528,1531],{"class":273,"line":274},[271,1527,278],{"class":277},[271,1529,1530],{"class":281}," save_query_locally",[271,1532,1382],{"class":285},[271,1534,1535,1537,1539,1541,1543,1545,1547],{"class":273,"line":187},[271,1536,1387],{"class":277},[271,1538,1390],{"class":281},[271,1540,1393],{"class":285},[271,1542,1396],{"class":277},[271,1544,1399],{"class":285},[271,1546,1402],{"class":277},[271,1548,1405],{"class":285},[271,1550,1551],{"class":273,"line":193},[271,1552,1553],{"class":345},"        # Before function call\n",[271,1555,1556,1558,1560,1562,1564],{"class":273,"line":324},[271,1557,1422],{"class":285},[271,1559,1396],{"class":277},[271,1561,1399],{"class":285},[271,1563,1402],{"class":277},[271,1565,1431],{"class":285},[271,1567,1568],{"class":273,"line":331},[271,1569,1570],{"class":345},"        # After function call\n",[271,1572,1573],{"class":273,"line":349},[271,1574,1575],{"class":345},"        # Save query to a local file\n",[271,1577,1578,1581,1583,1586,1588],{"class":273,"line":382},[271,1579,1580],{"class":285},"        notebook ",[271,1582,294],{"class":277},[271,1584,1585],{"class":285}," args[",[271,1587,1212],{"class":320},[271,1589,705],{"class":285},[271,1591,1592,1595,1597,1599,1601],{"class":273,"line":396},[271,1593,1594],{"class":285},"        user ",[271,1596,294],{"class":277},[271,1598,1585],{"class":285},[271,1600,816],{"class":320},[271,1602,705],{"class":285},[271,1604,1605,1608,1610,1612,1614,1616,1618,1620,1622],{"class":273,"line":417},[271,1606,1607],{"class":285},"        query_name ",[271,1609,294],{"class":277},[271,1611,700],{"class":285},[271,1613,443],{"class":300},[271,1615,1121],{"class":285},[271,1617,1124],{"class":300},[271,1619,304],{"class":285},[271,1621,1129],{"class":300},[271,1623,310],{"class":285},[271,1625,1626,1629,1631,1633,1635,1637],{"class":273,"line":426},[271,1627,1628],{"class":285},"        query_type ",[271,1630,294],{"class":277},[271,1632,700],{"class":285},[271,1634,301],{"class":300},[271,1636,611],{"class":285},[271,1638,1147],{"class":345},[271,1640,1641,1644,1646,1648,1650,1652,1654,1656,1658,1660],{"class":273,"line":476},[271,1642,1643],{"class":285},"        destination ",[271,1645,294],{"class":277},[271,1647,1157],{"class":300},[271,1649,851],{"class":320},[271,1651,1162],{"class":300},[271,1653,851],{"class":320},[271,1655,1162],{"class":300},[271,1657,851],{"class":320},[271,1659,1171],{"class":300},[271,1661,1662],{"class":285},".format(user, query_type, query_name)\n",[271,1664,1665,1668,1670,1672,1674,1676],{"class":273,"line":512},[271,1666,1667],{"class":285},"        output_file ",[271,1669,294],{"class":277},[271,1671,1184],{"class":320},[271,1673,1187],{"class":285},[271,1675,1190],{"class":300},[271,1677,310],{"class":285},[271,1679,1680,1683,1685,1687,1689,1691,1693,1695,1697],{"class":273,"line":523},[271,1681,1682],{"class":285},"        output_file.write(",[271,1684,1200],{"class":320},[271,1686,1203],{"class":285},[271,1688,1206],{"class":300},[271,1690,1209],{"class":285},[271,1692,1212],{"class":320},[271,1694,1209],{"class":285},[271,1696,1217],{"class":300},[271,1698,1220],{"class":285},[271,1700,1701],{"class":273,"line":528},[271,1702,1703],{"class":285},"        output_file.close()\n",[271,1705,1706,1708],{"class":273,"line":541},[271,1707,1447],{"class":277},[271,1709,1450],{"class":285},[14,1711,1712,1713,1716],{},"We copy this file inside the ",[268,1714,1715],{},"desktop\u002Flibs\u002Fnotebook\u002Fsrc\u002Fnotebook\u002F"," directory of the Hue installation.",[14,1718,1719,1720,1723],{},"And in the ",[268,1721,1722],{},"api.py"," file, first we import the newly created decorator by adding a new import statement",[263,1725,1727],{"className":265,"code":1726,"language":201,"meta":186,"style":186},"from notebook.query_decorators import save_query_locally\n",[268,1728,1729],{"__ignoreMap":186},[271,1730,1731,1734,1737,1740],{"class":273,"line":274},[271,1732,1733],{"class":277},"from",[271,1735,1736],{"class":285}," notebook.query_decorators ",[271,1738,1739],{"class":277},"import",[271,1741,1742],{"class":285}," save_query_locally\n",[14,1744,1745,1746,1748],{},"we then decorate the ",[268,1747,988],{}," function:",[263,1750,1752],{"className":265,"code":1751,"language":201,"meta":186,"style":186},"@save_query_locally\ndef _save_notebook(notebook, user):\n    # ...\n",[268,1753,1754,1759,1767],{"__ignoreMap":186},[271,1755,1756],{"class":273,"line":274},[271,1757,1758],{"class":281},"@save_query_locally\n",[271,1760,1761,1763,1765],{"class":273,"line":187},[271,1762,278],{"class":277},[271,1764,282],{"class":281},[271,1766,286],{"class":285},[271,1768,1769],{"class":273,"line":193},[271,1770,1771],{"class":345},"    # ...\n",[14,1773,1774],{},"Everytime a user saves a query, the new decorator will also take care of saving the query to a local file.",[14,1776,1777],{},"If needed the decorator can even go further by pushing the query to automatically version it, imagination is the limit.",[42,1779,1780],{},[14,1781,1782,1785,1786,1789,1790,1793],{},[70,1783,1784],{},"N.B."," All these steps can be automated, even by a simple bash script, for example by using ",[268,1787,1788],{},"cp"," or ",[268,1791,1792],{},"sed"," commands.",[27,1795,1797],{"id":1796},"conclusions","Conclusions",[14,1799,1800],{},"Decorators are quite a powerful Python feature that allows to achieve a lot of goals when writing your code, including extending product features without heavily modifying codebases that you cannot directly change or maintain.",[27,1802,1804],{"id":1803},"useful-resources","Useful Resources",[14,1806,1807,1808,1813,1814,1819,1820,1825],{},"If you want to learn more about decorators, you could see ",[219,1809,1812],{"href":1810,"rel":1811},"https:\u002F\u002Fwww.python.org\u002Fdev\u002Fpeps\u002Fpep-0318\u002F",[223],"PEP 318"," and the ",[219,1815,1818],{"href":1816,"rel":1817},"https:\u002F\u002Fwiki.python.org\u002Fmoin\u002FPythonDecorators",[223],"Python Decorator Wiki",", they both contain historical discussions and guidelines on how decorators should be implemented. You can find some examples of decorators in the ",[219,1821,1824],{"href":1822,"rel":1823},"https:\u002F\u002Fwiki.python.org\u002Fmoin\u002FPythonDecoratorLibrary",[223],"Python Decorator Library",".",[1827,1828,1829],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":186,"searchDepth":187,"depth":187,"links":1831},[1832,1833,1834,1835,1836,1837,1838],{"id":213,"depth":187,"text":214},{"id":237,"depth":187,"text":238},{"id":1003,"depth":187,"text":1004},{"id":1345,"depth":187,"text":1346},{"id":1507,"depth":187,"text":1508},{"id":1796,"depth":187,"text":1797},{"id":1803,"depth":187,"text":1804},{"path":1840,"title":1841,"created":1842,"tags":1843,"excerpt":1846,"body":1851},"\u002Fposts\u002Fa-gentle-introduction-to-functional-programming","A Gentle Introduction to Functional Programming","2019-08-25",[1844,1845],"scala","functional-programming",{"type":11,"value":1847},[1848],[14,1849,1850],{},"Having always worked with imperative programming languages, when I decided to switch my career towards Big Data technologies I had never heard of functional programming. At the beginning it was a shock.",{"type":11,"value":1852,"toc":2412},[1853,1855,1858,1861,1865,1874,1879,1882,1893,1896,1900,1906,1925,1928,1936,1939,1944,1947,1950,1953,2013,2021,2024,2108,2115,2120,2130,2133,2136,2140,2143,2146,2181,2184,2272,2275,2280,2293,2333,2349,2369,2373,2376,2381,2384,2387,2389,2392,2395,2398,2400,2409],[14,1854,1850],{},[14,1856,1857],{},"Functional programming, or for brevity FP, is becoming considerably relevant nowadays, as different typically imperative languages started integrating a few functional constructs in their syntax.",[14,1859,1860],{},"In this article I will try to define what is functional programming and provide a basic overview of the key concepts behind it.\nIn my sample code I will be using the Scala programming language.",[27,1862,1864],{"id":1863},"what-is-functional-programming","What is functional programming?",[14,1866,1867,1868,1873],{},"The definition of ",[219,1869,1872],{"href":1870,"rel":1871},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FFunctional_programming",[223],"functional programming"," on Wikipedia says:",[42,1875,1876],{},[14,1877,1878],{},"Functional programming is a programming paradigm that treats computation as the evaluation of mathematical functions and avoids changing-state and mutable data. It is a declarative programming paradigm in that programming is done with expressions or declarations instead of statements.",[14,1880,1881],{},"This definition states that FP has a few important characteristics:",[104,1883,1884,1887,1890],{},[107,1885,1886],{},"computation consists in evaluating mathematical functions",[107,1888,1889],{},"data is immutable",[107,1891,1892],{},"functional programming is declarative",[14,1894,1895],{},"The next sections will dive into each of the traits listed above, providing further details about it, showing the important implications they have on functional programming.",[27,1897,1899],{"id":1898},"computation-consists-in-evaluating-mathematical-functions","Computation consists in evaluating mathematical functions",[14,1901,1902,1903,1825],{},"One might argue that even imperative programming is about writing functions that are evaluated during computation, but the important keyword here is ",[70,1904,1905],{},"mathematical",[14,1907,1908,1909,1912,1913,1916,1917,1920,1921,1924],{},"In mathematics, given a function ",[268,1910,1911],{},"f"," and its input ",[268,1914,1915],{},"x",", it is true that ",[268,1918,1919],{},"f(x) = y",".\nThe resulting output, ",[268,1922,1923],{},"y",", only depends on how the input and the function are defined.",[14,1926,1927],{},"In functional programming, these mathematical functions are called \"pure functions\".\nThe output of a pure function only depends on",[104,1929,1930,1933],{},[107,1931,1932],{},"its input argument(s)",[107,1934,1935],{},"the internal algorithm of the function",[14,1937,1938],{},"This statement has two important implications:",[14,1940,1941],{},[70,1942,1943],{},"A pure function cannot contain side effects",[14,1945,1946],{},"A side effect consists in reading from or writing to the \"world\" outside the function.",[14,1948,1949],{},"Pure functions cannot change their return value or any of their properties after checking an hypothetical property in the global state of the program.\nThis also means that pure functions should not alter the global state of the program or the system, like by writing data inside a file.",[14,1951,1952],{},"This is a pure sum function in Scala:",[263,1954,1957],{"className":1955,"code":1956,"language":1844,"meta":186,"style":186},"language-scala shiki shiki-themes github-light github-dark","def sumNumbers(a: Int, b: Int): Int = {\n    a + b\n}\n",[268,1958,1959,1998,2009],{"__ignoreMap":186},[271,1960,1961,1963,1966,1968,1970,1972,1975,1977,1980,1982,1984,1987,1989,1992,1995],{"class":273,"line":274},[271,1962,278],{"class":277},[271,1964,1965],{"class":281}," sumNumbers",[271,1967,1393],{"class":285},[271,1969,219],{"class":360},[271,1971,810],{"class":285},[271,1973,1974],{"class":281},"Int",[271,1976,304],{"class":285},[271,1978,1979],{"class":360},"b",[271,1981,810],{"class":285},[271,1983,1974],{"class":281},[271,1985,1986],{"class":285},")",[271,1988,261],{"class":277},[271,1990,1991],{"class":281}," Int",[271,1993,1994],{"class":277}," =",[271,1996,1997],{"class":285}," {\n",[271,1999,2000,2003,2006],{"class":273,"line":187},[271,2001,2002],{"class":285},"    a ",[271,2004,2005],{"class":277},"+",[271,2007,2008],{"class":285}," b\n",[271,2010,2011],{"class":273,"line":193},[271,2012,819],{"class":285},[14,2014,2015,2016,2018,2019,1825],{},"Notice how the code is self-explaining itself: this function takes two numbers of ",[268,2017,1974],{}," type and returns their sum, another ",[268,2020,1974],{},[14,2022,2023],{},"What happens to this simple function if you add a side effect?",[263,2025,2027],{"className":1955,"code":2026,"language":1844,"meta":186,"style":186},"def sumNumbersImpure(a: Int, b: Int): Int = {\n    if (world.numRecords % 2 == 0) a + b else a + b + 1\n}\n",[268,2028,2029,2062,2104],{"__ignoreMap":186},[271,2030,2031,2033,2036,2038,2040,2042,2044,2046,2048,2050,2052,2054,2056,2058,2060],{"class":273,"line":274},[271,2032,278],{"class":277},[271,2034,2035],{"class":281}," sumNumbersImpure",[271,2037,1393],{"class":285},[271,2039,219],{"class":360},[271,2041,810],{"class":285},[271,2043,1974],{"class":281},[271,2045,304],{"class":285},[271,2047,1979],{"class":360},[271,2049,810],{"class":285},[271,2051,1974],{"class":281},[271,2053,1986],{"class":285},[271,2055,261],{"class":277},[271,2057,1991],{"class":281},[271,2059,1994],{"class":277},[271,2061,1997],{"class":285},[271,2063,2064,2066,2069,2072,2075,2078,2081,2084,2086,2089,2092,2095,2097,2099,2101],{"class":273,"line":187},[271,2065,531],{"class":277},[271,2067,2068],{"class":285}," (world.numRecords ",[271,2070,2071],{"class":277},"%",[271,2073,2074],{"class":320}," 2",[271,2076,2077],{"class":277}," ==",[271,2079,2080],{"class":320}," 0",[271,2082,2083],{"class":285},") a ",[271,2085,2005],{"class":277},[271,2087,2088],{"class":285}," b ",[271,2090,2091],{"class":277},"else",[271,2093,2094],{"class":285}," a ",[271,2096,2005],{"class":277},[271,2098,2088],{"class":285},[271,2100,2005],{"class":277},[271,2102,2103],{"class":320}," 1\n",[271,2105,2106],{"class":273,"line":193},[271,2107,819],{"class":285},[14,2109,2110,2111,2114],{},"The output of the function is unpredictable due to the statement ",[268,2112,2113],{},"world.numRecords % 2 == 0",", hence the function is not pure.",[14,2116,2117],{},[70,2118,2119],{},"A pure function always returns the same result for the same input",[14,2121,2122,2123,2126,2127,1825],{},"Given that the output of a pure function only depends on its input and implementation, and given that pure functions cannot have \"external dependencies\", a pure function will always return the same output for the same input. For instance, ",[268,2124,2125],{},"sumNumbers(3,2)"," will always return ",[268,2128,2129],{},"5",[14,2131,2132],{},"These pure functions properties will generate code that is quite reliable, other than easier to test and maintain.",[14,2134,2135],{},"In practice, a program can never be fully functional, but it is important that your code is structured as a strong FP core covered by an external layer of impure functions (write your data on the file system, update a system configuration, etc.). Communication between such layers is usually handled by an event bus or a queue-like structure.",[27,2137,2139],{"id":2138},"data-is-immutable","Data is immutable",[14,2141,2142],{},"Another cornerstone of FP is that data is immutable. Whenever a value is assigned to a named variable, that assignment is permanent.",[14,2144,2145],{},"In Scala this means that if you try to reassign a value to a variable, the code will throw an exception.",[263,2147,2149],{"className":1955,"code":2148,"language":1844,"meta":186,"style":186},"val a: Int = 42\na = 10 \u002F\u002F Will throw an error\n",[268,2150,2151,2168],{"__ignoreMap":186},[271,2152,2153,2156,2159,2161,2163,2165],{"class":273,"line":274},[271,2154,2155],{"class":277},"val",[271,2157,2158],{"class":360}," a",[271,2160,261],{"class":277},[271,2162,1991],{"class":281},[271,2164,1994],{"class":277},[271,2166,2167],{"class":320}," 42\n",[271,2169,2170,2173,2175,2178],{"class":273,"line":187},[271,2171,2172],{"class":285},"a ",[271,2174,294],{"class":277},[271,2176,2177],{"class":320}," 10",[271,2179,2180],{"class":345}," \u002F\u002F Will throw an error\n",[14,2182,2183],{},"The same happens when a property of a class or a data structure is updated. The solution is to create a new instance with the updated value for that field:",[263,2185,2187],{"className":1955,"code":2186,"language":1844,"meta":186,"style":186},"case class Person(firstName: String, lastName: String)\n\nval person = new Person(\"John\", \"Smith\")\nval personUpdatedName = new Person(\"Joe\", person.lastName)\n",[268,2188,2189,2221,2225,2251],{"__ignoreMap":186},[271,2190,2191,2194,2197,2200,2202,2205,2207,2210,2212,2215,2217,2219],{"class":273,"line":274},[271,2192,2193],{"class":277},"case",[271,2195,2196],{"class":277}," class",[271,2198,2199],{"class":281}," Person",[271,2201,1393],{"class":285},[271,2203,2204],{"class":360},"firstName",[271,2206,810],{"class":285},[271,2208,2209],{"class":281},"String",[271,2211,304],{"class":285},[271,2213,2214],{"class":360},"lastName",[271,2216,810],{"class":285},[271,2218,2209],{"class":281},[271,2220,310],{"class":285},[271,2222,2223],{"class":273,"line":187},[271,2224,328],{"emptyLinePlaceholder":327},[271,2226,2227,2229,2232,2234,2237,2239,2241,2244,2246,2249],{"class":273,"line":193},[271,2228,2155],{"class":277},[271,2230,2231],{"class":360}," person",[271,2233,1994],{"class":277},[271,2235,2236],{"class":277}," new",[271,2238,2199],{"class":281},[271,2240,1393],{"class":285},[271,2242,2243],{"class":300},"\"John\"",[271,2245,304],{"class":285},[271,2247,2248],{"class":300},"\"Smith\"",[271,2250,310],{"class":285},[271,2252,2253,2255,2258,2260,2262,2264,2266,2269],{"class":273,"line":324},[271,2254,2155],{"class":277},[271,2256,2257],{"class":360}," personUpdatedName",[271,2259,1994],{"class":277},[271,2261,2236],{"class":277},[271,2263,2199],{"class":281},[271,2265,1393],{"class":285},[271,2267,2268],{"class":300},"\"Joe\"",[271,2270,2271],{"class":285},", person.lastName)\n",[14,2273,2274],{},"The reason behind such choice is derived from the first property of FP:",[42,2276,2277],{},[14,2278,2279],{},"computation = evaluation of pure functions",[14,2281,2282,2283,2285,2286,2285,2289,2292],{},"Effective functional code is like algebra: variables are never reused.\nGiven three functions ",[268,2284,1911],{},",",[268,2287,2288],{},"g",[268,2290,2291],{},"h"," these two snippets are equivalent:",[263,2294,2296],{"className":1955,"code":2295,"language":1844,"meta":186,"style":186},"val a = f(x)\nval b = g(a)\nval c = h(b)\n",[268,2297,2298,2309,2321],{"__ignoreMap":186},[271,2299,2300,2302,2304,2306],{"class":273,"line":274},[271,2301,2155],{"class":277},[271,2303,2158],{"class":360},[271,2305,1994],{"class":277},[271,2307,2308],{"class":285}," f(x)\n",[271,2310,2311,2313,2316,2318],{"class":273,"line":187},[271,2312,2155],{"class":277},[271,2314,2315],{"class":360}," b",[271,2317,1994],{"class":277},[271,2319,2320],{"class":285}," g(a)\n",[271,2322,2323,2325,2328,2330],{"class":273,"line":193},[271,2324,2155],{"class":277},[271,2326,2327],{"class":360}," c",[271,2329,1994],{"class":277},[271,2331,2332],{"class":285}," h(b)\n",[263,2334,2336],{"className":1955,"code":2335,"language":1844,"meta":186,"style":186},"val c = h(g(f(x)))\n",[268,2337,2338],{"__ignoreMap":186},[271,2339,2340,2342,2344,2346],{"class":273,"line":274},[271,2341,2155],{"class":277},[271,2343,2327],{"class":360},[271,2345,1994],{"class":277},[271,2347,2348],{"class":285}," h(g(f(x)))\n",[14,2350,2351,2352,2355,2356,2359,2360,1789,2362,2364,2365,2368],{},"In functional programming, it is desirable that ",[268,2353,2354],{},"h(b)"," can always be replaced by ",[268,2357,2358],{},"h(g(f(x)))",". If ",[268,2361,219],{},[268,2363,1979],{}," were mutable, the equation would not always be true, making the functions impure, therefore preventing us to ",[268,2366,2367],{},"compose"," our functions calls.",[27,2370,2372],{"id":2371},"fp-is-declarative","FP is declarative",[14,2374,2375],{},"Functional programming is declarative. What does it mean?",[42,2377,2378],{},[14,2379,2380],{},"A declarative paradigm tells the program which tasks to perform, not how to perform them.",[14,2382,2383],{},"Two declarative languages are, for instance, SQL and RegExp: the first tells which data to return, the latter simply tells which patterns to match. In both cases the user will not know nor decide how to implement those tasks.",[14,2385,2386],{},"A functional program can be considered declarative, due to the computation consisting of input data and a sequence of functions that can be applied to it.\nThese functions should be considered like black boxes that given any input will return an output. In an imperative program you would have used constructs like for loops, that would have told the program how to produce the output.",[27,2388,1797],{"id":1796},[14,2390,2391],{},"This post is only scratching the surface of functional programming. There are still disparate topics to cover, plus plenty of FP jargon to demystify.",[14,2393,2394],{},"My hope is anyway that after reading this article you will have various questions and some curiosity on the subject, considering that functional programming might sound like the complete opposite of how almost all programmers were taught.",[14,2396,2397],{},"Learning this new paradigm will turn you into a programmer inclined to produce code that is simple, reliable and testable, despite the programming language you are using.",[27,2399,1804],{"id":1803},[14,2401,2402,2403,2408],{},"If you are interested in functional programming and are just a beginner, but also if you want to study in deep some specific topics, I cannot recommend any better book than ",[219,2404,2407],{"href":2405,"rel":2406},"https:\u002F\u002Fgeni.us\u002FMJaf",[223],"Functional Programming Simplified"," by Alvin Alexander.",[1827,2410,2411],{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}",{"title":186,"searchDepth":187,"depth":187,"links":2413},[2414,2415,2416,2417,2418,2419],{"id":1863,"depth":187,"text":1864},{"id":1898,"depth":187,"text":1899},{"id":2138,"depth":187,"text":2139},{"id":2371,"depth":187,"text":2372},{"id":1796,"depth":187,"text":1797},{"id":1803,"depth":187,"text":1804},1788801177195]