User:Matthew.zellmer/bots

From Lotro-Wiki.com
Jump to navigation Jump to search

I want to make Bots that will allow me to find/replace the same text on multiple pages. I can do this using the amazing API, PHP and cURL.


Wiki API can be used over HTTP (internet) to access the inside of the wiki: it can query and edit the wiki

API action=query

To get more information distilled from the wikitext at parse time (links, categories, sections, etc.), use the API parse module: https://en.wikipedia.org/w/api.php?action=parse&page=Main_Page . See also the documentation for the API:Parsing wikitext#parse action=parse module.

API action=edit

  • Changing wiki content using the action=edit to edit a page
  • three-step process, query to get current data and get edit token, modify data locally, POST data (API:Edit) back to server
    • perform query and get token happen at the same time (use intoken=edit in the query)
    • API tokens can only be asked for using specific queries prop=info&titles=FindThisExactTitle is one of them
    • manipulate the data locally on your computer
    • POST back to wiki, POST requests only: Diff between GET and POST
    • POST and PHP
    • Only way to send POST data is to use a web form
    • So you have to have a local webpage that will accept the action=query data into a form then use the form to POST it back to the wiki
    • This is what the python scripts do but they do most of the heavy lifting for you
    • read the API documentation about action=edit here

Bots and Robots access the API to perform actions

My solution: Use HTML forms, PHP, JavaScript and AJAX to access the API

  • 1. HTML form to build a GET query, use AJAX to send it to the API and get a XML response
  • 2. format the returned response, so that I can select what to keep or throw out
  • 3. perform a very specific GET query on each selected item and get an edit token at the same time
  • 4. search the returned response and add/edit what I want locally using JavaScript or PHP
  • 5. send a POST edit action back to the server with the changes using the edit token received earlier

AJAX fail

Use cURL and PHP Instead of AJAX

Robot, Bot, Scripts, Completed, Solution

  • AJAX has limited use given the cross-domain restrictions, I cant get around them
  • use other coding options

Build PHP scripts to perform API query only

  • design:
    • PHP webpage with textbox and form, must know API query language yourself, limited help
    • form submits to PHP backend, which performs API query, makes a local copy of the returned XML results
    • new page displays the results but you can checkmark the ones you want for further API query
    • ended up using the PHP cURL for GET/POST requests to the server
  • DONE - its sloppy but its done

Built a PHP script that use cURL to perform API login

  • used to login and hold local information for future API edits
  • required for any future API POST requests
  • design:
    • PHP website with a textbox and form to hold the API login request (URL)
    • form submits a PHP cURL POST login request
    • grab the returned login token (number)
    • send the login again with the token using the PHP cURL POST
    • verify login by displaying the outcome
  • DONE, a bit of a slob on the coding

Build a PHP script that use cURL to perform API edit on a single place

  • design:
    • use the test website https://testwiki.skunark.net/index.php
    • perform API query to search for text
    • use initial API query and make a checkbox to allow user a chance to CANCEL future edit
    • have a form that will accept, find/replace text on the selected pages (textboxes), submit button
      • perform API query GET to get full page text on a selected page and edit token
      • find and replace text
      • displace old and new versions of the page text
      • pop-up box/alert box to allow the user one last ditch change to CANCEL
      • perform API edit POST submitting the entire page text (with new replaced text)
  • DONE - again sloppy but done, no error handling

Build a PHP script that use cURL to perform API edit on multiple pages

  • design:
    • use the test website https://testwiki.skunark.net/index.php
    • perform API query to search for text, return multiple results
    • use initial API query to make a list that user can select (checkbox) pages from a previous query
    • have a form that will accept, find/replace text on the selected pages (textboxes), submit button
      • perform API query GET to get full page text on a selected page and edit token
      • find and replace text
      • displace old and new versions of the page text
      • perform API edit POST submitting the entire page text (with new replaced text)
  • DONE, but its sloppy, query doesn't work right in finding exact pages yet either

Build a PHP script that use cURL to perform API edit on multiple pages

  • design:
    • use the test website https://testwiki.skunark.net/index.php
    • perform API query to search for text, return multiple results
    • use initial API query to make a list that user can select (checkbox) pages from a previous query
    • have a form that will accept, find/replace text on the selected pages (textboxes), submit button
      • perform API query GET to get full page text on a selected page and edit token
      • find and replace text
      • displace old and new versions of the page text
      • perform API edit POST submitting the entire page text (with new replaced text)
  • DONE, but its sloppy, query doesn't work right in finding exact pages yet either

Make my scripts able to perform multiple search/find/replace for each page to be edited

  • design:
    • use previous scripts with single find/replace and modify them
    • create more find/replace text boxes
    • loop thru each find and replace text instead of just the single find/replace text
    • output success for each find/replace or output NO CHANGES MADE
    • only execute the API edit if changes were made
    • display results of API edit if performed
    • removed the textarea that displays the entire pages with changes (its just to much to review)
  • DONE! even added a tiny bit of debugging code, but for the most part it crashes terribly still

Make my scripts able to perform search/find and delete entire line for each page to be edited

  • design:
    • use previous scripts with single find/replace and modify
    • create a form checkbox that, if checked will delete the entire line if something on it matched the find
    • output success for each line deleted or output NO CHANGES MADE
    • only execute the API edit if changes were made
    • display results of API edit if performed
  • DONE!

test wiki site

  • test scripts here
  • testwiki.skunark.net
  • lotroadmin will have to give you bot rights on that.
  • I also don't know the status of the test-wiki currently, as it was "munged" as part of the Server switch/upgrade last month.(Dec2013)
Retrieved from "https://lotro-wiki.com/w/index.php?title=User:Matthew.zellmer/bots&oldid=1702358"